Jul 16 08:52:52.105086 ip-10-0-128-59 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 08:52:52.105099 ip-10-0-128-59 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 08:52:52.105108 ip-10-0-128-59 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 08:52:52.105575 ip-10-0-128-59 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 08:53:01.497503 ip-10-0-128-59 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 16 08:53:01.497519 ip-10-0-128-59 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5f0896bb90e140e980740357b97da9cb -- Jul 16 08:54:46.519691 ip-10-0-128-59 systemd[1]: Starting Kubernetes Kubelet... Jul 16 08:54:46.870427 ip-10-0-128-59 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 08:54:46.870427 ip-10-0-128-59 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 08:54:46.870427 ip-10-0-128-59 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 08:54:46.870427 ip-10-0-128-59 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 08:54:46.870427 ip-10-0-128-59 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 08:54:46.871311 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.871078 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 08:54:46.875685 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875654 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875694 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875699 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875704 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875709 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875713 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875716 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875720 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875724 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875727 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875730 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875733 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875736 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875740 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 08:54:46.875734 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875743 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875746 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875750 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875753 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875756 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875759 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875762 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875776 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875779 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875782 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875786 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875789 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875791 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875794 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875797 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875800 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875803 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875806 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875809 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875812 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 08:54:46.876112 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875814 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875817 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875820 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875823 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875826 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875828 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875831 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875835 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875837 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875840 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875844 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875847 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875850 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875852 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875856 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875859 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875862 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875865 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875868 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 08:54:46.876648 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875871 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875891 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875894 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875897 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875900 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875903 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875906 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875911 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875914 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875917 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875920 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875923 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875926 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875929 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875932 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875936 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875940 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875944 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875948 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875952 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 08:54:46.877265 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875956 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875959 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875961 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875964 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875967 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875970 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875973 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875976 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875979 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875982 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875987 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875991 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875996 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.875999 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876009 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876012 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876016 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876019 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876022 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876025 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 08:54:46.878054 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876028 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876031 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876034 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876037 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876040 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.876043 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878059 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878082 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878109 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878118 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878125 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878130 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878136 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878141 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878147 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878152 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878157 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878180 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878188 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 08:54:46.878737 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878194 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878206 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878211 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878217 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878222 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878227 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878232 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878236 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878243 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878248 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878253 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878259 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878264 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878275 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878279 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878284 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878289 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878293 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878332 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878338 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 08:54:46.879277 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878343 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878348 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878354 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878362 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878368 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878378 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878383 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878389 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878394 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878399 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878406 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878434 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878488 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878501 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878526 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878657 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878664 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878668 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878672 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878677 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 08:54:46.879805 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878681 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878686 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878690 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878695 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878699 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878704 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878708 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878712 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878717 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878721 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878729 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878735 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878741 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878746 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878751 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878764 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878769 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878773 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878777 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 08:54:46.880403 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878782 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878786 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878790 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878795 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878799 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878804 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878814 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878820 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878825 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878829 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878834 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878838 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878842 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878847 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878851 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878856 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878861 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878866 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878871 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 08:54:46.880888 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878875 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878878 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878880 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878883 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878887 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878891 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878894 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.878897 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879454 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879464 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879468 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879472 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879475 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879479 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879483 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879486 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879488 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879491 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879495 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879498 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 08:54:46.881419 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879501 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879504 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879507 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879510 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879513 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879516 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879519 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879522 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879525 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879527 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879531 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879534 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879537 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879540 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879542 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879545 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879548 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879550 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879553 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879556 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 08:54:46.881936 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879559 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879562 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879565 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879567 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879570 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879573 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879575 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879578 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879581 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879584 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879587 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879590 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879593 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879596 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879599 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879602 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879605 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879609 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879612 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879614 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 08:54:46.882483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879617 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879620 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879623 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879627 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879629 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879633 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879637 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879641 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879644 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879647 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879650 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879652 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879657 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879661 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879664 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879668 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879671 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879676 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879679 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 08:54:46.882988 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879683 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879686 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879689 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879692 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879696 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879699 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879702 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879705 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879708 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879711 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879714 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879717 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879720 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879723 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879726 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879729 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879732 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879735 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879738 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879741 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 08:54:46.883506 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879744 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879746 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879749 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879752 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879755 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879758 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879762 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.879765 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879861 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879876 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879886 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879893 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879900 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879904 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879909 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879914 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879917 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879921 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879925 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879929 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879933 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879936 2574 flags.go:64] FLAG: --cgroup-root="" Jul 16 08:54:46.884141 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879939 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879942 2574 flags.go:64] FLAG: --client-ca-file="" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879947 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879950 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879957 2574 flags.go:64] FLAG: --cluster-domain="" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879960 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879964 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879967 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879971 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879975 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879978 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879982 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879985 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879989 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879993 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.879996 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880000 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880004 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880009 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880013 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880016 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880020 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880026 2574 flags.go:64] FLAG: --enable-server="true" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880031 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880037 2574 flags.go:64] FLAG: --event-burst="100" Jul 16 08:54:46.884794 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880041 2574 flags.go:64] FLAG: --event-qps="50" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880044 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880047 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880050 2574 flags.go:64] FLAG: --eviction-hard="" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880055 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880058 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880061 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880064 2574 flags.go:64] FLAG: --eviction-soft="" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880068 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880071 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880075 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880079 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880086 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880090 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880093 2574 flags.go:64] FLAG: --feature-gates="" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880097 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880101 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880104 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880107 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880111 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880115 2574 flags.go:64] FLAG: --help="false" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880121 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-128-59.ec2.internal" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880124 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880127 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 08:54:46.885450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880130 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880134 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880138 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880142 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880145 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880148 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880152 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880155 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880159 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880161 2574 flags.go:64] FLAG: --kube-reserved="" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880165 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880168 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880171 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880175 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880180 2574 flags.go:64] FLAG: --lock-file="" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880185 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880190 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880194 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880222 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880244 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880250 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880253 2574 flags.go:64] FLAG: --logging-format="text" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880256 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880260 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 08:54:46.886102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880263 2574 flags.go:64] FLAG: --manifest-url="" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880266 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880272 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880275 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880280 2574 flags.go:64] FLAG: --max-pods="110" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880285 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880288 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880292 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880295 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880298 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880302 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880305 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880314 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880317 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880321 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880324 2574 flags.go:64] FLAG: --pod-cidr="" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880327 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880333 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880336 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880340 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880343 2574 flags.go:64] FLAG: --port="10250" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880347 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880350 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a19508e084d151df" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880354 2574 flags.go:64] FLAG: --qos-reserved="" Jul 16 08:54:46.886741 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880357 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880360 2574 flags.go:64] FLAG: --register-node="true" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880364 2574 flags.go:64] FLAG: --register-with-taints="" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880368 2574 flags.go:64] FLAG: --registry-burst="10" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880372 2574 flags.go:64] FLAG: --registry-qps="5" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880377 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880381 2574 flags.go:64] FLAG: --reserved-memory="" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880385 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880388 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880392 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880395 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880398 2574 flags.go:64] FLAG: --runonce="false" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880402 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880407 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880410 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880414 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880417 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880421 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880424 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880428 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880431 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880435 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880438 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880441 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880444 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 08:54:46.887401 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880447 2574 flags.go:64] FLAG: --system-cgroups="" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880450 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880456 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880460 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880463 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880468 2574 flags.go:64] FLAG: --tls-min-version="" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880471 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880474 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880478 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880481 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880484 2574 flags.go:64] FLAG: --v="2" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880488 2574 flags.go:64] FLAG: --version="false" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880495 2574 flags.go:64] FLAG: --vmodule="" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880500 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.880505 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880650 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880655 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880658 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880662 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880665 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880670 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880673 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880677 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 08:54:46.888065 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880680 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880682 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880686 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880688 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880691 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880694 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880698 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880700 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880703 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880706 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880709 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880712 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880715 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880718 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880721 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880724 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880728 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880733 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880736 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 08:54:46.888738 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880739 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880741 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880746 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880750 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880753 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880756 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880759 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880762 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880765 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880768 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880773 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880776 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880779 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880781 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880784 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880787 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880790 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880793 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880796 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880799 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 08:54:46.889297 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880802 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880804 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880807 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880810 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880813 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880816 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880818 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880821 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880824 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880827 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880831 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880835 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880839 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880843 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880847 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880850 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880854 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880857 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880860 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 08:54:46.889817 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880863 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880866 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880869 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880873 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880877 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880879 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880883 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880887 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880890 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880894 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880897 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880901 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880904 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880907 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880911 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880914 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880917 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880920 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880923 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880927 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 08:54:46.890380 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880930 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880933 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880936 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880939 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880942 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880945 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880948 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880952 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880955 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880958 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880961 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880964 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.880967 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.881479 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.889675 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.889703 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 08:54:46.890914 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.890433 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.890442 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890516 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890523 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890527 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890531 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890534 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890537 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890541 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890544 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890549 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890554 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890558 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890561 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890565 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890568 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890571 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890575 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890578 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 08:54:46.891366 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890581 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890585 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890589 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890592 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890595 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890598 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890602 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890606 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890610 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890613 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890616 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890619 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890623 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890626 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890629 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890632 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890635 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890637 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890640 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 08:54:46.891854 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890644 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890647 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890650 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890653 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890655 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890659 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890662 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890665 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890668 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890671 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890673 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890676 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890679 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890682 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890685 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890688 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890690 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890693 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890696 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890699 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 08:54:46.892387 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890702 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890706 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890709 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890712 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890716 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890720 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890723 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890726 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890729 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890731 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890735 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890739 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890742 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890746 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890749 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890752 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890756 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890760 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890763 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890766 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 08:54:46.892904 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890769 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890772 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890775 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890779 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890782 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890785 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890787 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890792 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890796 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890799 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890802 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890805 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890807 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890810 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890814 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890818 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890821 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890825 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890828 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 08:54:46.893462 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890831 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890835 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890837 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890840 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.890847 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890990 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890996 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.890999 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891003 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891006 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891010 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891013 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891016 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891019 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891022 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 08:54:46.893953 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891025 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891028 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891031 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891034 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891036 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891039 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891042 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891045 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891048 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891050 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891053 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891056 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891059 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891062 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891066 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891069 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891073 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891076 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891079 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891084 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 08:54:46.894452 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891088 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891091 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891094 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891097 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891100 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891103 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891106 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891109 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891112 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891114 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891117 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891120 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891123 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891126 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891129 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891132 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891134 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891137 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891140 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 08:54:46.894966 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891143 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891146 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891151 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891154 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891157 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891160 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891163 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891168 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891173 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891175 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891178 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891181 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891184 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891187 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891190 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891192 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891195 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891198 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891200 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 08:54:46.895483 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891203 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891206 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891209 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891226 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891230 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891233 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891236 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891239 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891242 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891245 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891247 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891250 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891253 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891256 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891260 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891263 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891266 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891269 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891272 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 08:54:46.895980 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891275 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891278 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891282 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891285 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891288 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891291 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891294 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891297 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891299 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891302 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891305 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:46.891308 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.891314 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.891480 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 08:54:46.896551 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.894853 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 08:54:46.896932 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.896076 2574 server.go:1019] "Starting client certificate rotation" Jul 16 08:54:46.896932 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.896201 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 08:54:46.896932 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.896241 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 08:54:46.898425 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.898406 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 08:54:46.899834 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.899811 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 08:54:46.908317 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.908291 2574 log.go:25] "Validated CRI v1 runtime API" Jul 16 08:54:46.910689 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.910667 2574 log.go:25] "Validated CRI v1 image API" Jul 16 08:54:46.912133 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.912110 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 08:54:46.914584 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.914559 2574 fs.go:135] Filesystem UUIDs: map[1849cf6d-fcfc-466e-a982-b36a61f9ce54:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8410db95-f67f-4895-9e5f-46da4b118396:/dev/nvme0n1p3] Jul 16 08:54:46.914660 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.914587 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 08:54:46.921930 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.921763 2574 manager.go:216] Machine: {Timestamp:2026-07-16 08:54:46.920593792 +0000 UTC m=+0.294728076 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099838 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ea86ceb9037d70990fc2a7e7f5b2 SystemUUID:ec23ea86-ceb9-037d-7099-0fc2a7e7f5b2 BootID:5f0896bb-90e1-40e9-8074-0357b97da9cb Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0f:a8:59:f7:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0f:a8:59:f7:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:00:87:05:57:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 08:54:46.921930 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.921904 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 08:54:46.922150 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.922024 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 08:54:46.923787 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.923743 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 08:54:46.923995 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.923788 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-59.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 08:54:46.924526 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.924513 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 08:54:46.924584 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.924532 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 08:54:46.924584 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.924543 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 08:54:46.925102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.925088 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 08:54:46.925936 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.925921 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 08:54:46.928381 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.928356 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 08:54:46.930692 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.930673 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 16 08:54:46.930884 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.930871 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 08:54:46.930947 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.930919 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 16 08:54:46.930947 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.930939 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 08:54:46.931042 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.930960 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 08:54:46.931042 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.930977 2574 kubelet.go:398] "Adding apiserver pod source" Jul 16 08:54:46.931042 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.930991 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 08:54:46.931525 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.931508 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 08:54:46.931604 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.931533 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 08:54:46.934126 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.934097 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 08:54:46.935688 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.935667 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 08:54:46.935804 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.935710 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 08:54:46.936412 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936396 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 08:54:46.936485 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936420 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 08:54:46.936485 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936431 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 08:54:46.936485 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936440 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 08:54:46.936485 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936454 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 08:54:46.936485 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936464 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 08:54:46.936485 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936476 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 08:54:46.936485 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936485 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 08:54:46.936742 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936498 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 08:54:46.936742 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936507 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 08:54:46.936742 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936522 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 08:54:46.936742 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.936531 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 08:54:46.937199 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.937185 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 08:54:46.937274 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.937201 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 08:54:46.941164 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.941143 2574 server.go:1268] "Started kubelet" Jul 16 08:54:46.941293 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.941251 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 08:54:46.941358 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.941283 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 08:54:46.941400 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.941376 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 08:54:46.942267 ip-10-0-128-59 systemd[1]: Started Kubernetes Kubelet. Jul 16 08:54:46.942490 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:46.942463 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 08:54:46.942564 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.942485 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-59.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 08:54:46.942621 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:46.942564 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 08:54:46.943106 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.943085 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 08:54:46.944019 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.944003 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 16 08:54:46.947022 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.947002 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 08:54:46.947123 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.947031 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 08:54:46.947195 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:46.947127 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:46.947655 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.947619 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 08:54:46.947655 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.947624 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 08:54:46.947804 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.947665 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 08:54:46.947868 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.947808 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 16 08:54:46.947868 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.947815 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 16 08:54:46.948344 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.948266 2574 factory.go:55] Registering systemd factory Jul 16 08:54:46.948344 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.948288 2574 factory.go:223] Registration of the systemd container factory successfully Jul 16 08:54:46.948603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.948579 2574 factory.go:153] Registering CRI-O factory Jul 16 08:54:46.948603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.948603 2574 factory.go:223] Registration of the crio container factory successfully Jul 16 08:54:46.948777 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.948688 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 08:54:46.948777 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.948749 2574 factory.go:103] Registering Raw factory Jul 16 08:54:46.948777 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.948774 2574 manager.go:1196] Started watching for new ooms in manager Jul 16 08:54:46.949598 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:46.949574 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 08:54:46.949692 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.949635 2574 manager.go:349] Starting recovery of all containers Jul 16 08:54:46.950793 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:46.950763 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 08:54:46.950995 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:46.950940 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-59.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 08:54:46.952160 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:46.951224 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-59.ec2.internal.18c2b94001300ba3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-59.ec2.internal,UID:ip-10-0-128-59.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-59.ec2.internal,},FirstTimestamp:2026-07-16 08:54:46.941109155 +0000 UTC m=+0.315243543,LastTimestamp:2026-07-16 08:54:46.941109155 +0000 UTC m=+0.315243543,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-59.ec2.internal,}" Jul 16 08:54:46.963587 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.963557 2574 manager.go:354] Recovery completed Jul 16 08:54:46.964084 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.964059 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pgsw2" Jul 16 08:54:46.968719 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.968703 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 08:54:46.971187 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.971164 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory" Jul 16 08:54:46.971313 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.971262 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 08:54:46.971313 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.971285 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID" Jul 16 08:54:46.971982 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.971964 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 08:54:46.971982 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.971981 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 08:54:46.972118 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.971985 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 08:54:46.972522 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.972510 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 16 08:54:46.973002 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.972969 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory" Jul 16 08:54:46.973108 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.973015 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 08:54:46.973108 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.973047 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID" Jul 16 08:54:46.973372 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:46.973203 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-59.ec2.internal.18c2b94002fbd81b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-59.ec2.internal,UID:ip-10-0-128-59.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-59.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-59.ec2.internal,},FirstTimestamp:2026-07-16 08:54:46.971242523 +0000 UTC m=+0.345376810,LastTimestamp:2026-07-16 08:54:46.971242523 +0000 UTC m=+0.345376810,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-59.ec2.internal,}" Jul 16 08:54:46.974961 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.974946 2574 policy_none.go:49] "None policy: Start" Jul 16 08:54:46.974961 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.974967 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 08:54:46.975092 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.974979 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 08:54:46.975130 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.975095 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pgsw2" Jul 16 08:54:46.977285 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:46.977272 2574 policy_none.go:47] "Start" Jul 16 08:54:47.011130 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.011099 2574 manager.go:337] "Starting Device Plugin manager" Jul 16 08:54:47.011311 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.011150 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 08:54:47.011311 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.011163 2574 server.go:87] "Starting device plugin registration server" Jul 16 08:54:47.011604 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.011582 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 08:54:47.011728 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.011600 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 08:54:47.011728 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.011681 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 08:54:47.011854 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.011774 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 08:54:47.011854 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.011783 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 08:54:47.012547 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.012516 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 08:54:47.012633 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.012595 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.047845 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.047780 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 08:54:47.055148 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.049305 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 08:54:47.055148 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.049350 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 08:54:47.055148 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.049378 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 08:54:47.055148 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.049418 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 08:54:47.055148 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.054073 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 08:54:47.114078 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.114036 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 08:54:47.115302 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.115276 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory" Jul 16 08:54:47.115423 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.115319 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 08:54:47.115423 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.115331 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID" Jul 16 08:54:47.115423 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.115361 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.124258 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.124190 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.124258 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.124229 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-59.ec2.internal\": node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.138596 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.138562 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.149646 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.149609 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal"] Jul 16 08:54:47.149748 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.149704 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 08:54:47.153966 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.153945 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory" Jul 16 08:54:47.154065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.153983 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 08:54:47.154065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.154001 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID" Jul 16 08:54:47.156800 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.156780 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 08:54:47.156923 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.156906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.156992 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.156945 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 08:54:47.157756 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.157735 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory" Jul 16 08:54:47.157756 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.157749 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory" Jul 16 08:54:47.157881 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.157775 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 08:54:47.157881 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.157788 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID" Jul 16 08:54:47.157881 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.157846 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 08:54:47.157881 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.157862 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID" Jul 16 08:54:47.160483 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.160463 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.160595 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.160496 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 08:54:47.161453 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.161432 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientMemory" Jul 16 08:54:47.161535 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.161473 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 08:54:47.161535 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.161489 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeHasSufficientPID" Jul 16 08:54:47.178074 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.178039 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-59.ec2.internal\" not found" node="ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.182962 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.182938 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-59.ec2.internal\" not found" node="ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.239262 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.239218 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.248950 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.248924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6462d2c44e021ea56ae54eb1c6c365ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"6462d2c44e021ea56ae54eb1c6c365ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.249072 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.248957 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6462d2c44e021ea56ae54eb1c6c365ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"6462d2c44e021ea56ae54eb1c6c365ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.249072 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.248985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dfbfc1bca38cfe8c6a6944c393e430e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-59.ec2.internal\" (UID: \"8dfbfc1bca38cfe8c6a6944c393e430e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.339397 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.339341 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.350132 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.350101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6462d2c44e021ea56ae54eb1c6c365ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"6462d2c44e021ea56ae54eb1c6c365ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.350248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.350137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dfbfc1bca38cfe8c6a6944c393e430e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-59.ec2.internal\" (UID: \"8dfbfc1bca38cfe8c6a6944c393e430e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.350248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.350205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6462d2c44e021ea56ae54eb1c6c365ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"6462d2c44e021ea56ae54eb1c6c365ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.350248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.350226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6462d2c44e021ea56ae54eb1c6c365ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"6462d2c44e021ea56ae54eb1c6c365ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.350367 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.350246 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8dfbfc1bca38cfe8c6a6944c393e430e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-59.ec2.internal\" (UID: \"8dfbfc1bca38cfe8c6a6944c393e430e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.350367 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.350256 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6462d2c44e021ea56ae54eb1c6c365ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal\" (UID: \"6462d2c44e021ea56ae54eb1c6c365ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.440459 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.440354 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.481899 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.481850 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.485751 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.485726 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.541298 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.541249 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.648654 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.648583 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.749738 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.749646 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.850763 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.850729 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-59.ec2.internal\" not found" Jul 16 08:54:47.896983 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.896945 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 08:54:47.897692 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.897165 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 08:54:47.907065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.907042 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 08:54:47.931723 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.931675 2574 apiserver.go:52] "Watching apiserver" Jul 16 08:54:47.940916 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.940887 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 08:54:47.942533 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.942505 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-pd2px","openshift-image-registry/node-ca-lpps8","openshift-multus/multus-additional-cni-plugins-2zqbr","openshift-network-diagnostics/network-check-target-kbb9c","openshift-ovn-kubernetes/ovnkube-node-lf9r4","kube-system/konnectivity-agent-njtsp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf","openshift-cluster-node-tuning-operator/tuned-xtljz","openshift-dns/node-resolver-qhwlv","openshift-multus/multus-6j42v","openshift-multus/network-metrics-daemon-7zblf","openshift-network-operator/iptables-alerter-vwvrj"] Jul 16 08:54:47.945428 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.945403 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:47.945582 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.945500 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:54:47.947767 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.947736 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.947913 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.947774 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:47.947913 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.947793 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 08:54:47.950659 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.950281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:47.950659 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.950407 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 08:54:47.950659 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.950441 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 08:54:47.950920 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.950762 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kdsms\"" Jul 16 08:54:47.950920 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.950883 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 08:54:47.952626 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.952603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:47.952751 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.952664 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:54:47.952751 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.952607 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 08:54:47.952851 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.952757 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 08:54:47.953254 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.953237 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 08:54:47.953345 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.953323 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 08:54:47.953345 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.953337 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 08:54:47.953455 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.953399 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v5qpc\"" Jul 16 08:54:47.954670 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-system-cni-dir\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:47.954754 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-cnibin\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:47.954754 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vccbf\" (UniqueName: \"kubernetes.io/projected/9dac8b01-f667-4faa-801b-c69c9655c829-kube-api-access-vccbf\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:47.954754 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/928dda30-aac8-42e9-a376-9eb125067780-host\") pod \"node-ca-lpps8\" (UID: \"928dda30-aac8-42e9-a376-9eb125067780\") " pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:47.954894 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2fg\" (UniqueName: \"kubernetes.io/projected/928dda30-aac8-42e9-a376-9eb125067780-kube-api-access-8t2fg\") pod \"node-ca-lpps8\" (UID: \"928dda30-aac8-42e9-a376-9eb125067780\") " pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:47.954894 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954800 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9dac8b01-f667-4faa-801b-c69c9655c829-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:47.954894 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/491e030c-547f-4c64-be98-eb0c8313b800-dbus\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:47.954894 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/928dda30-aac8-42e9-a376-9eb125067780-serviceca\") pod \"node-ca-lpps8\" (UID: \"928dda30-aac8-42e9-a376-9eb125067780\") " pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:47.955092 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954904 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:47.955092 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9dac8b01-f667-4faa-801b-c69c9655c829-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:47.955092 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9dac8b01-f667-4faa-801b-c69c9655c829-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:47.955092 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.954980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/491e030c-547f-4c64-be98-eb0c8313b800-kubelet-config\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:47.955092 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.955042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-os-release\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:47.955092 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.955064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:47.955329 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.955188 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:47.957378 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.957358 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 08:54:47.957498 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.957479 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 08:54:47.957607 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.957588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:54:47.958089 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.958072 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 08:54:47.958182 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.958094 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-msdnz\"" Jul 16 08:54:47.958182 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.958147 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 08:54:47.958182 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.958094 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 08:54:47.959838 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.959820 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 08:54:47.960050 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.960018 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 08:54:47.960121 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.960076 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 08:54:47.960570 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.960537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:47.960661 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.960592 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zw8jd\"" Jul 16 08:54:47.961755 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.961724 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 08:54:47.963821 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.962754 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 08:54:47.963821 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.962942 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" Jul 16 08:54:47.963821 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.963385 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 08:54:47.963821 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.963732 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 08:54:47.964295 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.964275 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pqvpg\"" Jul 16 08:54:47.964455 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.964435 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 08:54:47.965486 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.965464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:47.967868 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.967850 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 08:54:47.967969 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.967854 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jfcl2\"" Jul 16 08:54:47.968127 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.968110 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:47.968843 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.968826 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 08:54:47.970595 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.970575 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6j42v" Jul 16 08:54:47.970809 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.970790 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m5hqm\"" Jul 16 08:54:47.971178 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.971163 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 08:54:47.971248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.971216 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 08:54:47.972991 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.972971 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 08:54:47.973292 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.973271 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 08:54:47.973419 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.973304 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5k9zn\"" Jul 16 08:54:47.973671 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.973655 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:47.973751 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:47.973722 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:54:47.976108 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.976086 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal"] Jul 16 08:54:47.976254 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.976124 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal"] Jul 16 08:54:47.976254 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.976220 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:47.979855 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.979809 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 08:49:47 +0000 UTC" deadline="2028-02-10 15:25:52.416028642 +0000 UTC" Jul 16 08:54:47.979940 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.979854 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13782h31m4.436177804s" Jul 16 08:54:47.979940 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.979916 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 08:54:47.980022 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.979916 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 08:54:47.981414 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.981398 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72l49\"" Jul 16 08:54:47.985468 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.985448 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 08:54:47.994282 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:47.994254 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f45fb" Jul 16 08:54:48.005494 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.005403 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f45fb" Jul 16 08:54:48.049067 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.049039 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 08:54:48.055502 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-cnibin\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.055669 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-etc-openvswitch\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.055669 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-sysctl-conf\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.055669 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7b6480a-2e28-4325-b994-4a1bd5516125-cni-binary-copy\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.055669 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9dac8b01-f667-4faa-801b-c69c9655c829-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.055669 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-sysconfig\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.055669 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9dac8b01-f667-4faa-801b-c69c9655c829-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.055931 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.055931 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04273c03-411a-4deb-b522-061973f7d4a7-etc-tuned\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.055931 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-cni-dir\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.055931 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055765 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-systemd-units\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.055931 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-log-socket\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.055931 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jdd\" (UniqueName: \"kubernetes.io/projected/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-kube-api-access-r4jdd\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:48.055931 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:48.055931 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-host\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.056291 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.055933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-os-release\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.056291 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-socket-dir-parent\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.056291 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-var-lib-kubelet\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.056291 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vccbf\" (UniqueName: \"kubernetes.io/projected/9dac8b01-f667-4faa-801b-c69c9655c829-kube-api-access-vccbf\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.056291 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6fl\" (UniqueName: \"kubernetes.io/projected/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-kube-api-access-8s6fl\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.056291 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c64883f-3ead-4052-bea8-c09675421988-tmp-dir\") pod \"node-resolver-qhwlv\" (UID: \"0c64883f-3ead-4052-bea8-c09675421988\") " pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zcfj\" (UniqueName: \"kubernetes.io/projected/04273c03-411a-4deb-b522-061973f7d4a7-kube-api-access-5zcfj\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-etc-kubernetes\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-run-openvswitch\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056407 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9dac8b01-f667-4faa-801b-c69c9655c829-cni-binary-copy\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056422 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-var-lib-cni-multus\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9dac8b01-f667-4faa-801b-c69c9655c829-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056478 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9dac8b01-f667-4faa-801b-c69c9655c829-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-run-ovn\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-sys-fs\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnzt\" (UniqueName: \"kubernetes.io/projected/c7b6480a-2e28-4325-b994-4a1bd5516125-kube-api-access-jrnzt\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.056603 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-os-release\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-var-lib-openvswitch\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvmff\" (UniqueName: \"kubernetes.io/projected/08a75a91-2b97-4b16-8951-8d91c167dea5-kube-api-access-pvmff\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-systemd\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056700 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-os-release\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-run-k8s-cni-cncf-io\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056750 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd03ff02-e36a-47a4-b9bb-16b91534c524-iptables-alerter-script\") pod \"iptables-alerter-vwvrj\" (UID: \"bd03ff02-e36a-47a4-b9bb-16b91534c524\") " pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-system-cni-dir\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/928dda30-aac8-42e9-a376-9eb125067780-host\") pod \"node-ca-lpps8\" (UID: \"928dda30-aac8-42e9-a376-9eb125067780\") " pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-system-cni-dir\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056895 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/928dda30-aac8-42e9-a376-9eb125067780-host\") pod \"node-ca-lpps8\" (UID: \"928dda30-aac8-42e9-a376-9eb125067780\") " pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2fg\" (UniqueName: \"kubernetes.io/projected/928dda30-aac8-42e9-a376-9eb125067780-kube-api-access-8t2fg\") pod \"node-ca-lpps8\" (UID: \"928dda30-aac8-42e9-a376-9eb125067780\") " pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08a75a91-2b97-4b16-8951-8d91c167dea5-ovnkube-script-lib\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9dac8b01-f667-4faa-801b-c69c9655c829-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.056983 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-socket-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn7nb\" (UniqueName: \"kubernetes.io/projected/0c64883f-3ead-4052-bea8-c09675421988-kube-api-access-cn7nb\") pod \"node-resolver-qhwlv\" (UID: \"0c64883f-3ead-4052-bea8-c09675421988\") " pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057056 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-sys\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.057114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057105 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-lib-modules\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/928dda30-aac8-42e9-a376-9eb125067780-serviceca\") pod \"node-ca-lpps8\" (UID: \"928dda30-aac8-42e9-a376-9eb125067780\") " pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-run-systemd\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08a75a91-2b97-4b16-8951-8d91c167dea5-ovnkube-config\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/180ba7da-0767-42a2-9f01-4492fff3b648-agent-certs\") pod \"konnectivity-agent-njtsp\" (UID: \"180ba7da-0767-42a2-9f01-4492fff3b648\") " pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-run\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-run-netns\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057362 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd03ff02-e36a-47a4-b9bb-16b91534c524-host-slash\") pod \"iptables-alerter-vwvrj\" (UID: \"bd03ff02-e36a-47a4-b9bb-16b91534c524\") " pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/491e030c-547f-4c64-be98-eb0c8313b800-kubelet-config\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-kubelet\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057438 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/491e030c-547f-4c64-be98-eb0c8313b800-kubelet-config\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08a75a91-2b97-4b16-8951-8d91c167dea5-env-overrides\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04273c03-411a-4deb-b522-061973f7d4a7-tmp\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cczq6\" (UniqueName: \"kubernetes.io/projected/bd03ff02-e36a-47a4-b9bb-16b91534c524-kube-api-access-cczq6\") pod \"iptables-alerter-vwvrj\" (UID: \"bd03ff02-e36a-47a4-b9bb-16b91534c524\") " pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057591 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/928dda30-aac8-42e9-a376-9eb125067780-serviceca\") pod \"node-ca-lpps8\" (UID: \"928dda30-aac8-42e9-a376-9eb125067780\") " pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:48.057825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-cni-bin\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.057626 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/180ba7da-0767-42a2-9f01-4492fff3b648-konnectivity-ca\") pod \"konnectivity-agent-njtsp\" (UID: \"180ba7da-0767-42a2-9f01-4492fff3b648\") " pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-system-cni-dir\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.057706 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret podName:491e030c-547f-4c64-be98-eb0c8313b800 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:48.557671815 +0000 UTC m=+1.931806089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret") pod "global-pull-secret-syncer-pd2px" (UID: "491e030c-547f-4c64-be98-eb0c8313b800") : object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-hostroot\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057776 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-conf-dir\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-daemon-config\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-cnibin\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-run-netns\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057909 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-cnibin\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-cni-netd\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-device-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.057988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058013 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c64883f-3ead-4052-bea8-c09675421988-hosts-file\") pod \"node-resolver-qhwlv\" (UID: \"0c64883f-3ead-4052-bea8-c09675421988\") " pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/491e030c-547f-4c64-be98-eb0c8313b800-dbus\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:48.058643 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-slash\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08a75a91-2b97-4b16-8951-8d91c167dea5-ovn-node-metrics-cert\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-registration-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/491e030c-547f-4c64-be98-eb0c8313b800-dbus\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-modprobe-d\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058341 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-node-log\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058370 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-run-ovn-kubernetes\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058401 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-kubernetes\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-var-lib-kubelet\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058461 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-var-lib-cni-bin\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-run-multus-certs\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058522 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9dac8b01-f667-4faa-801b-c69c9655c829-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.059170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.058546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-sysctl-d\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.063377 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.063350 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 08:54:48.067407 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.067369 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vccbf\" (UniqueName: \"kubernetes.io/projected/9dac8b01-f667-4faa-801b-c69c9655c829-kube-api-access-vccbf\") pod \"multus-additional-cni-plugins-2zqbr\" (UID: \"9dac8b01-f667-4faa-801b-c69c9655c829\") " pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.067837 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.067298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2fg\" (UniqueName: \"kubernetes.io/projected/928dda30-aac8-42e9-a376-9eb125067780-kube-api-access-8t2fg\") pod \"node-ca-lpps8\" (UID: \"928dda30-aac8-42e9-a376-9eb125067780\") " pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:48.159356 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-slash\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.159356 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08a75a91-2b97-4b16-8951-8d91c167dea5-ovn-node-metrics-cert\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-registration-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-modprobe-d\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-node-log\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159463 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-slash\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-run-ovn-kubernetes\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159485 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159513 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-node-log\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-kubernetes\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159514 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-registration-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159554 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-run-ovn-kubernetes\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-var-lib-kubelet\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-kubernetes\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.159609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-var-lib-cni-bin\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-run-multus-certs\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159636 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-modprobe-d\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159637 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-var-lib-kubelet\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-sysctl-d\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-cnibin\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-run-multus-certs\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159665 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-var-lib-cni-bin\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-etc-openvswitch\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-cnibin\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159773 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-sysctl-d\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-sysctl-conf\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159814 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7b6480a-2e28-4325-b994-4a1bd5516125-cni-binary-copy\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-etc-openvswitch\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-sysconfig\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04273c03-411a-4deb-b522-061973f7d4a7-etc-tuned\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159913 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-sysctl-conf\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.160325 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159925 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-sysconfig\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.159954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-cni-dir\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-systemd-units\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-log-socket\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jdd\" (UniqueName: \"kubernetes.io/projected/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-kube-api-access-r4jdd\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-log-socket\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-host\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160132 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-cni-dir\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-os-release\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-systemd-units\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-host\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-os-release\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160272 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-socket-dir-parent\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-var-lib-kubelet\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160309 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-socket-dir-parent\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6fl\" (UniqueName: \"kubernetes.io/projected/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-kube-api-access-8s6fl\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.161222 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160363 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-var-lib-kubelet\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c64883f-3ead-4052-bea8-c09675421988-tmp-dir\") pod \"node-resolver-qhwlv\" (UID: \"0c64883f-3ead-4052-bea8-c09675421988\") " pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zcfj\" (UniqueName: \"kubernetes.io/projected/04273c03-411a-4deb-b522-061973f7d4a7-kube-api-access-5zcfj\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-etc-kubernetes\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-run-openvswitch\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-var-lib-cni-multus\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-run-ovn\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-sys-fs\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnzt\" (UniqueName: \"kubernetes.io/projected/c7b6480a-2e28-4325-b994-4a1bd5516125-kube-api-access-jrnzt\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-var-lib-openvswitch\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvmff\" (UniqueName: \"kubernetes.io/projected/08a75a91-2b97-4b16-8951-8d91c167dea5-kube-api-access-pvmff\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160634 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-sys-fs\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-etc-kubernetes\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160655 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c64883f-3ead-4052-bea8-c09675421988-tmp-dir\") pod \"node-resolver-qhwlv\" (UID: \"0c64883f-3ead-4052-bea8-c09675421988\") " pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160687 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-systemd\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160687 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-var-lib-openvswitch\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160612 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-var-lib-cni-multus\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-etc-systemd\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.162167 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-run-openvswitch\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-run-k8s-cni-cncf-io\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160747 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-run-ovn\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd03ff02-e36a-47a4-b9bb-16b91534c524-iptables-alerter-script\") pod \"iptables-alerter-vwvrj\" (UID: \"bd03ff02-e36a-47a4-b9bb-16b91534c524\") " pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08a75a91-2b97-4b16-8951-8d91c167dea5-ovnkube-script-lib\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-run-k8s-cni-cncf-io\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-socket-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7nb\" (UniqueName: \"kubernetes.io/projected/0c64883f-3ead-4052-bea8-c09675421988-kube-api-access-cn7nb\") pod \"node-resolver-qhwlv\" (UID: \"0c64883f-3ead-4052-bea8-c09675421988\") " pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-sys\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-lib-modules\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160951 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-run-systemd\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.160976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08a75a91-2b97-4b16-8951-8d91c167dea5-ovnkube-config\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7b6480a-2e28-4325-b994-4a1bd5516125-cni-binary-copy\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.161106 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161141 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-socket-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.161181 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs podName:7689dfa7-1f38-485d-a5e5-1fbea1c74f57 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:48.661159452 +0000 UTC m=+2.035293723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs") pod "network-metrics-daemon-7zblf" (UID: "7689dfa7-1f38-485d-a5e5-1fbea1c74f57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:48.162975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/180ba7da-0767-42a2-9f01-4492fff3b648-agent-certs\") pod \"konnectivity-agent-njtsp\" (UID: \"180ba7da-0767-42a2-9f01-4492fff3b648\") " pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161204 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-run-systemd\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-run\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161286 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-sys\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161306 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-run\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-run-netns\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd03ff02-e36a-47a4-b9bb-16b91534c524-host-slash\") pod \"iptables-alerter-vwvrj\" (UID: \"bd03ff02-e36a-47a4-b9bb-16b91534c524\") " pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161377 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-kubelet\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08a75a91-2b97-4b16-8951-8d91c167dea5-env-overrides\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd03ff02-e36a-47a4-b9bb-16b91534c524-iptables-alerter-script\") pod \"iptables-alerter-vwvrj\" (UID: \"bd03ff02-e36a-47a4-b9bb-16b91534c524\") " pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04273c03-411a-4deb-b522-061973f7d4a7-tmp\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-host-run-netns\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cczq6\" (UniqueName: \"kubernetes.io/projected/bd03ff02-e36a-47a4-b9bb-16b91534c524-kube-api-access-cczq6\") pod \"iptables-alerter-vwvrj\" (UID: \"bd03ff02-e36a-47a4-b9bb-16b91534c524\") " pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161494 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd03ff02-e36a-47a4-b9bb-16b91534c524-host-slash\") pod \"iptables-alerter-vwvrj\" (UID: \"bd03ff02-e36a-47a4-b9bb-16b91534c524\") " pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-cni-bin\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-kubelet\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/180ba7da-0767-42a2-9f01-4492fff3b648-konnectivity-ca\") pod \"konnectivity-agent-njtsp\" (UID: \"180ba7da-0767-42a2-9f01-4492fff3b648\") " pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/08a75a91-2b97-4b16-8951-8d91c167dea5-ovnkube-script-lib\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.163641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161585 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04273c03-411a-4deb-b522-061973f7d4a7-lib-modules\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-system-cni-dir\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-hostroot\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161644 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-system-cni-dir\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-conf-dir\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-daemon-config\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-run-netns\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-cni-netd\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161775 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-device-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c64883f-3ead-4052-bea8-c09675421988-hosts-file\") pod \"node-resolver-qhwlv\" (UID: \"0c64883f-3ead-4052-bea8-c09675421988\") " pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161919 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0c64883f-3ead-4052-bea8-c09675421988-hosts-file\") pod \"node-resolver-qhwlv\" (UID: \"0c64883f-3ead-4052-bea8-c09675421988\") " pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161966 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-run-netns\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.162037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-cni-netd\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.162030 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/08a75a91-2b97-4b16-8951-8d91c167dea5-ovnkube-config\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.162081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-device-dir\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.162117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-hostroot\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.161637 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/08a75a91-2b97-4b16-8951-8d91c167dea5-host-cni-bin\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.164376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.162183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.165000 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.162226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-conf-dir\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.165000 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.162273 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/180ba7da-0767-42a2-9f01-4492fff3b648-konnectivity-ca\") pod \"konnectivity-agent-njtsp\" (UID: \"180ba7da-0767-42a2-9f01-4492fff3b648\") " pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:54:48.165000 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.162866 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/08a75a91-2b97-4b16-8951-8d91c167dea5-env-overrides\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.165000 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.162903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7b6480a-2e28-4325-b994-4a1bd5516125-multus-daemon-config\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.165000 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.163143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/08a75a91-2b97-4b16-8951-8d91c167dea5-ovn-node-metrics-cert\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.165000 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.164655 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/04273c03-411a-4deb-b522-061973f7d4a7-etc-tuned\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.165000 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.164676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04273c03-411a-4deb-b522-061973f7d4a7-tmp\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.165000 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.164777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/180ba7da-0767-42a2-9f01-4492fff3b648-agent-certs\") pod \"konnectivity-agent-njtsp\" (UID: \"180ba7da-0767-42a2-9f01-4492fff3b648\") " pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:54:48.170035 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.170011 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 08:54:48.170035 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.170034 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 08:54:48.170376 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.170048 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ghljh for pod openshift-network-diagnostics/network-check-target-kbb9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:48.170376 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.170121 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh podName:e21baeba-d058-495d-81b1-7a93f16372c3 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:48.670103695 +0000 UTC m=+2.044237966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ghljh" (UniqueName: "kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh") pod "network-check-target-kbb9c" (UID: "e21baeba-d058-495d-81b1-7a93f16372c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:48.172430 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.172398 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zcfj\" (UniqueName: \"kubernetes.io/projected/04273c03-411a-4deb-b522-061973f7d4a7-kube-api-access-5zcfj\") pod \"tuned-xtljz\" (UID: \"04273c03-411a-4deb-b522-061973f7d4a7\") " pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.173774 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.173740 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7nb\" (UniqueName: \"kubernetes.io/projected/0c64883f-3ead-4052-bea8-c09675421988-kube-api-access-cn7nb\") pod \"node-resolver-qhwlv\" (UID: \"0c64883f-3ead-4052-bea8-c09675421988\") " pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.173947 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.173929 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cczq6\" (UniqueName: \"kubernetes.io/projected/bd03ff02-e36a-47a4-b9bb-16b91534c524-kube-api-access-cczq6\") pod \"iptables-alerter-vwvrj\" (UID: \"bd03ff02-e36a-47a4-b9bb-16b91534c524\") " pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.174030 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.174012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnzt\" (UniqueName: \"kubernetes.io/projected/c7b6480a-2e28-4325-b994-4a1bd5516125-kube-api-access-jrnzt\") pod \"multus-6j42v\" (UID: \"c7b6480a-2e28-4325-b994-4a1bd5516125\") " pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.174281 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.174257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvmff\" (UniqueName: \"kubernetes.io/projected/08a75a91-2b97-4b16-8951-8d91c167dea5-kube-api-access-pvmff\") pod \"ovnkube-node-lf9r4\" (UID: \"08a75a91-2b97-4b16-8951-8d91c167dea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.174478 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.174457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6fl\" (UniqueName: \"kubernetes.io/projected/bc6ee5a6-68ee-436a-99d0-46b4abd359fe-kube-api-access-8s6fl\") pod \"aws-ebs-csi-driver-node-q4lwf\" (UID: \"bc6ee5a6-68ee-436a-99d0-46b4abd359fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.174820 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.174803 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jdd\" (UniqueName: \"kubernetes.io/projected/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-kube-api-access-r4jdd\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:48.193932 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.193892 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfbfc1bca38cfe8c6a6944c393e430e.slice/crio-07f82d9373be0f235a22148d4392410e575d404604fc9daca5f065914e96c145 WatchSource:0}: Error finding container 07f82d9373be0f235a22148d4392410e575d404604fc9daca5f065914e96c145: Status 404 returned error can't find the container with id 07f82d9373be0f235a22148d4392410e575d404604fc9daca5f065914e96c145 Jul 16 08:54:48.194489 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.194466 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6462d2c44e021ea56ae54eb1c6c365ab.slice/crio-25ec7851bec68a30712484833d702a9246c76159a8d5cd1d1c671c81df847e15 WatchSource:0}: Error finding container 25ec7851bec68a30712484833d702a9246c76159a8d5cd1d1c671c81df847e15: Status 404 returned error can't find the container with id 25ec7851bec68a30712484833d702a9246c76159a8d5cd1d1c671c81df847e15 Jul 16 08:54:48.199201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.199178 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 08:54:48.277454 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.277335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lpps8" Jul 16 08:54:48.288248 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.288193 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928dda30_aac8_42e9_a376_9eb125067780.slice/crio-e6d647099c4fe2d9bc65114b5e829e13cea712110ea68350a4d7b32ccb5f4d19 WatchSource:0}: Error finding container e6d647099c4fe2d9bc65114b5e829e13cea712110ea68350a4d7b32ccb5f4d19: Status 404 returned error can't find the container with id e6d647099c4fe2d9bc65114b5e829e13cea712110ea68350a4d7b32ccb5f4d19 Jul 16 08:54:48.291764 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.291743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" Jul 16 08:54:48.302060 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.302030 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dac8b01_f667_4faa_801b_c69c9655c829.slice/crio-466f1d510c11c7d5e11bfa34e5b6c3b7a5ebd12bbb0cabfc602e7a69b430aa14 WatchSource:0}: Error finding container 466f1d510c11c7d5e11bfa34e5b6c3b7a5ebd12bbb0cabfc602e7a69b430aa14: Status 404 returned error can't find the container with id 466f1d510c11c7d5e11bfa34e5b6c3b7a5ebd12bbb0cabfc602e7a69b430aa14 Jul 16 08:54:48.314500 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.314458 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:54:48.324678 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.324642 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a75a91_2b97_4b16_8951_8d91c167dea5.slice/crio-c2034a8f8490c5ed87476145b05c6220dbb6a28dbfc9f600673d64b98291e0de WatchSource:0}: Error finding container c2034a8f8490c5ed87476145b05c6220dbb6a28dbfc9f600673d64b98291e0de: Status 404 returned error can't find the container with id c2034a8f8490c5ed87476145b05c6220dbb6a28dbfc9f600673d64b98291e0de Jul 16 08:54:48.326789 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.326763 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:54:48.335852 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.335821 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod180ba7da_0767_42a2_9f01_4492fff3b648.slice/crio-71475c6b1971c212cbffc14df0d5918dccc53835e36fd45cc863bd0307af22ff WatchSource:0}: Error finding container 71475c6b1971c212cbffc14df0d5918dccc53835e36fd45cc863bd0307af22ff: Status 404 returned error can't find the container with id 71475c6b1971c212cbffc14df0d5918dccc53835e36fd45cc863bd0307af22ff Jul 16 08:54:48.344205 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.344180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" Jul 16 08:54:48.353819 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.353783 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6ee5a6_68ee_436a_99d0_46b4abd359fe.slice/crio-6ae2f8b7c1124acbde01db49956cd2c99c1ab5dd456f5524ad86df89382fa37e WatchSource:0}: Error finding container 6ae2f8b7c1124acbde01db49956cd2c99c1ab5dd456f5524ad86df89382fa37e: Status 404 returned error can't find the container with id 6ae2f8b7c1124acbde01db49956cd2c99c1ab5dd456f5524ad86df89382fa37e Jul 16 08:54:48.355769 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.355752 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xtljz" Jul 16 08:54:48.365202 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.365174 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04273c03_411a_4deb_b522_061973f7d4a7.slice/crio-db9fbde5bc4cdc69ca106a05fc38ca8830169cde0f47806e1b49dce8e6c2a248 WatchSource:0}: Error finding container db9fbde5bc4cdc69ca106a05fc38ca8830169cde0f47806e1b49dce8e6c2a248: Status 404 returned error can't find the container with id db9fbde5bc4cdc69ca106a05fc38ca8830169cde0f47806e1b49dce8e6c2a248 Jul 16 08:54:48.375353 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.375326 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qhwlv" Jul 16 08:54:48.385500 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.385468 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c64883f_3ead_4052_bea8_c09675421988.slice/crio-c56e38358aa049ba429fd81da67995a060db8737821020c0f3e3c68ad7b885da WatchSource:0}: Error finding container c56e38358aa049ba429fd81da67995a060db8737821020c0f3e3c68ad7b885da: Status 404 returned error can't find the container with id c56e38358aa049ba429fd81da67995a060db8737821020c0f3e3c68ad7b885da Jul 16 08:54:48.394580 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.394550 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6j42v" Jul 16 08:54:48.400990 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.400959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vwvrj" Jul 16 08:54:48.405849 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.405815 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b6480a_2e28_4325_b994_4a1bd5516125.slice/crio-5a843acfa84c09808e702496c45cf0786f9b335ef16eade600a0ecd3f3fe6f96 WatchSource:0}: Error finding container 5a843acfa84c09808e702496c45cf0786f9b335ef16eade600a0ecd3f3fe6f96: Status 404 returned error can't find the container with id 5a843acfa84c09808e702496c45cf0786f9b335ef16eade600a0ecd3f3fe6f96 Jul 16 08:54:48.411066 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:54:48.411040 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd03ff02_e36a_47a4_b9bb_16b91534c524.slice/crio-8c137362fd049d8dd0644df27b40593d4467ac38ebdfa314231ee40040d7af69 WatchSource:0}: Error finding container 8c137362fd049d8dd0644df27b40593d4467ac38ebdfa314231ee40040d7af69: Status 404 returned error can't find the container with id 8c137362fd049d8dd0644df27b40593d4467ac38ebdfa314231ee40040d7af69 Jul 16 08:54:48.432324 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.432294 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 08:54:48.537246 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.537145 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 08:54:48.563273 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.563236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:48.563494 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.563363 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:48.563494 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.563422 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret podName:491e030c-547f-4c64-be98-eb0c8313b800 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:49.56340372 +0000 UTC m=+2.937537993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret") pod "global-pull-secret-syncer-pd2px" (UID: "491e030c-547f-4c64-be98-eb0c8313b800") : object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:48.664460 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.664419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:48.664661 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.664597 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:48.664735 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.664669 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs podName:7689dfa7-1f38-485d-a5e5-1fbea1c74f57 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:49.664650981 +0000 UTC m=+3.038785254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs") pod "network-metrics-daemon-7zblf" (UID: "7689dfa7-1f38-485d-a5e5-1fbea1c74f57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:48.765158 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:48.765116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:48.765403 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.765342 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 08:54:48.765403 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.765364 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 08:54:48.765403 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.765381 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ghljh for pod openshift-network-diagnostics/network-check-target-kbb9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:48.765608 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:48.765449 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh podName:e21baeba-d058-495d-81b1-7a93f16372c3 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:49.765425781 +0000 UTC m=+3.139560053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghljh" (UniqueName: "kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh") pod "network-check-target-kbb9c" (UID: "e21baeba-d058-495d-81b1-7a93f16372c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:49.009600 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.009518 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 08:49:48 +0000 UTC" deadline="2028-01-30 16:22:12.444945607 +0000 UTC" Jul 16 08:54:49.009600 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.009598 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13519h27m23.435352796s" Jul 16 08:54:49.052946 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.052917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:49.053141 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:49.053066 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:54:49.075072 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.075001 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xtljz" event={"ID":"04273c03-411a-4deb-b522-061973f7d4a7","Type":"ContainerStarted","Data":"db9fbde5bc4cdc69ca106a05fc38ca8830169cde0f47806e1b49dce8e6c2a248"} Jul 16 08:54:49.095443 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.095397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" event={"ID":"bc6ee5a6-68ee-436a-99d0-46b4abd359fe","Type":"ContainerStarted","Data":"6ae2f8b7c1124acbde01db49956cd2c99c1ab5dd456f5524ad86df89382fa37e"} Jul 16 08:54:49.116891 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.116846 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-njtsp" event={"ID":"180ba7da-0767-42a2-9f01-4492fff3b648","Type":"ContainerStarted","Data":"71475c6b1971c212cbffc14df0d5918dccc53835e36fd45cc863bd0307af22ff"} Jul 16 08:54:49.128158 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.128005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerStarted","Data":"c2034a8f8490c5ed87476145b05c6220dbb6a28dbfc9f600673d64b98291e0de"} Jul 16 08:54:49.149745 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.149477 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" event={"ID":"9dac8b01-f667-4faa-801b-c69c9655c829","Type":"ContainerStarted","Data":"466f1d510c11c7d5e11bfa34e5b6c3b7a5ebd12bbb0cabfc602e7a69b430aa14"} Jul 16 08:54:49.168357 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.168241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lpps8" event={"ID":"928dda30-aac8-42e9-a376-9eb125067780","Type":"ContainerStarted","Data":"e6d647099c4fe2d9bc65114b5e829e13cea712110ea68350a4d7b32ccb5f4d19"} Jul 16 08:54:49.197352 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.194112 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" event={"ID":"6462d2c44e021ea56ae54eb1c6c365ab","Type":"ContainerStarted","Data":"25ec7851bec68a30712484833d702a9246c76159a8d5cd1d1c671c81df847e15"} Jul 16 08:54:49.212484 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.212438 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vwvrj" event={"ID":"bd03ff02-e36a-47a4-b9bb-16b91534c524","Type":"ContainerStarted","Data":"8c137362fd049d8dd0644df27b40593d4467ac38ebdfa314231ee40040d7af69"} Jul 16 08:54:49.222488 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.222441 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6j42v" event={"ID":"c7b6480a-2e28-4325-b994-4a1bd5516125","Type":"ContainerStarted","Data":"5a843acfa84c09808e702496c45cf0786f9b335ef16eade600a0ecd3f3fe6f96"} Jul 16 08:54:49.237490 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.237379 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" event={"ID":"8dfbfc1bca38cfe8c6a6944c393e430e","Type":"ContainerStarted","Data":"07f82d9373be0f235a22148d4392410e575d404604fc9daca5f065914e96c145"} Jul 16 08:54:49.248641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.247103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qhwlv" event={"ID":"0c64883f-3ead-4052-bea8-c09675421988","Type":"ContainerStarted","Data":"c56e38358aa049ba429fd81da67995a060db8737821020c0f3e3c68ad7b885da"} Jul 16 08:54:49.428178 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.428137 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 08:54:49.573656 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.573599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:49.573882 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:49.573783 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:49.573882 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:49.573864 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret podName:491e030c-547f-4c64-be98-eb0c8313b800 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:51.573842025 +0000 UTC m=+4.947976298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret") pod "global-pull-secret-syncer-pd2px" (UID: "491e030c-547f-4c64-be98-eb0c8313b800") : object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:49.674670 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.674624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:49.674881 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:49.674793 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:49.674881 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:49.674879 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs podName:7689dfa7-1f38-485d-a5e5-1fbea1c74f57 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:51.674856793 +0000 UTC m=+5.048991078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs") pod "network-metrics-daemon-7zblf" (UID: "7689dfa7-1f38-485d-a5e5-1fbea1c74f57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:49.775815 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:49.775552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:49.775815 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:49.775809 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 08:54:49.776103 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:49.775833 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 08:54:49.776103 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:49.775850 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ghljh for pod openshift-network-diagnostics/network-check-target-kbb9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:49.776103 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:49.775920 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh podName:e21baeba-d058-495d-81b1-7a93f16372c3 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:51.775899094 +0000 UTC m=+5.150033386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghljh" (UniqueName: "kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh") pod "network-check-target-kbb9c" (UID: "e21baeba-d058-495d-81b1-7a93f16372c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:50.009872 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:50.009743 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 08:49:48 +0000 UTC" deadline="2028-03-11 13:40:18.034781488 +0000 UTC" Jul 16 08:54:50.009872 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:50.009798 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14500h45m28.024989603s" Jul 16 08:54:50.050585 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:50.050545 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:50.050817 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:50.050707 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:54:50.051311 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:50.051287 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:50.052011 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:50.051440 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:54:51.053114 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:51.053080 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:51.053688 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:51.053256 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:54:51.595627 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:51.594835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:51.595627 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:51.595083 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:51.595627 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:51.595193 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret podName:491e030c-547f-4c64-be98-eb0c8313b800 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:55.595170727 +0000 UTC m=+8.969305001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret") pod "global-pull-secret-syncer-pd2px" (UID: "491e030c-547f-4c64-be98-eb0c8313b800") : object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:51.696337 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:51.696096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:51.697648 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:51.696962 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:51.697648 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:51.697070 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs podName:7689dfa7-1f38-485d-a5e5-1fbea1c74f57 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:55.697042701 +0000 UTC m=+9.071176987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs") pod "network-metrics-daemon-7zblf" (UID: "7689dfa7-1f38-485d-a5e5-1fbea1c74f57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:51.797626 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:51.797513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:51.797841 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:51.797717 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 08:54:51.797841 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:51.797745 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 08:54:51.797841 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:51.797764 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ghljh for pod openshift-network-diagnostics/network-check-target-kbb9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:51.798042 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:51.797853 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh podName:e21baeba-d058-495d-81b1-7a93f16372c3 nodeName:}" failed. No retries permitted until 2026-07-16 08:54:55.797831894 +0000 UTC m=+9.171966178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghljh" (UniqueName: "kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh") pod "network-check-target-kbb9c" (UID: "e21baeba-d058-495d-81b1-7a93f16372c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:52.050567 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:52.050532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:52.050797 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:52.050573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:52.050797 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:52.050741 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:54:52.050927 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:52.050886 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:54:53.050822 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:53.050257 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:53.050822 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:53.050438 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:54:54.049983 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:54.049931 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:54.049983 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:54.049977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:54.050295 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:54.050103 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:54:54.050295 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:54.050263 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:54:55.050667 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:55.050628 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:55.051258 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:55.050791 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:54:55.631721 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:55.631638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:55.631918 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:55.631817 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:55.631918 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:55.631899 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret podName:491e030c-547f-4c64-be98-eb0c8313b800 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:03.631875169 +0000 UTC m=+17.006009439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret") pod "global-pull-secret-syncer-pd2px" (UID: "491e030c-547f-4c64-be98-eb0c8313b800") : object "kube-system"/"original-pull-secret" not registered Jul 16 08:54:55.732901 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:55.732868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:55.733107 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:55.733052 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:55.733188 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:55.733133 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs podName:7689dfa7-1f38-485d-a5e5-1fbea1c74f57 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:03.733110832 +0000 UTC m=+17.107245112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs") pod "network-metrics-daemon-7zblf" (UID: "7689dfa7-1f38-485d-a5e5-1fbea1c74f57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:54:55.834533 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:55.834316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:55.834533 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:55.834526 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 08:54:55.834533 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:55.834547 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 08:54:55.834856 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:55.834561 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ghljh for pod openshift-network-diagnostics/network-check-target-kbb9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:55.834856 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:55.834632 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh podName:e21baeba-d058-495d-81b1-7a93f16372c3 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:03.834612711 +0000 UTC m=+17.208746982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghljh" (UniqueName: "kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh") pod "network-check-target-kbb9c" (UID: "e21baeba-d058-495d-81b1-7a93f16372c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:54:56.050030 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:56.049923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:56.050248 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:56.050076 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:54:56.050248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:56.050234 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:56.050373 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:56.050341 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:54:57.051515 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:57.051471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:57.052010 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:57.051635 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:54:58.050428 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:58.049971 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:54:58.050428 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:58.050404 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:54:58.050943 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:58.050023 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:54:58.050943 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:58.050898 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:54:59.049942 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:54:59.049709 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:54:59.049942 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:54:59.049871 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:00.050359 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:00.050320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:00.050851 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:00.050332 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:00.050851 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:00.050467 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:00.050851 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:00.050547 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:01.050482 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:01.050436 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:01.050927 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:01.050552 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:02.050655 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:02.050610 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:02.051195 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:02.050745 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:02.051195 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:02.050811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:02.051195 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:02.050943 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:03.050517 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:03.050479 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:03.050740 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:03.050619 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:03.697053 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:03.697010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:03.697279 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:03.697196 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 08:55:03.697336 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:03.697282 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret podName:491e030c-547f-4c64-be98-eb0c8313b800 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:19.697264465 +0000 UTC m=+33.071398737 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret") pod "global-pull-secret-syncer-pd2px" (UID: "491e030c-547f-4c64-be98-eb0c8313b800") : object "kube-system"/"original-pull-secret" not registered Jul 16 08:55:03.797548 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:03.797510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:03.797776 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:03.797665 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:55:03.797776 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:03.797737 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs podName:7689dfa7-1f38-485d-a5e5-1fbea1c74f57 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:19.797719951 +0000 UTC m=+33.171854222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs") pod "network-metrics-daemon-7zblf" (UID: "7689dfa7-1f38-485d-a5e5-1fbea1c74f57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:55:03.897985 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:03.897943 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:03.898212 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:03.898148 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 08:55:03.898212 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:03.898189 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 08:55:03.898212 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:03.898200 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ghljh for pod openshift-network-diagnostics/network-check-target-kbb9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:55:03.898359 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:03.898256 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh podName:e21baeba-d058-495d-81b1-7a93f16372c3 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:19.898241344 +0000 UTC m=+33.272375612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghljh" (UniqueName: "kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh") pod "network-check-target-kbb9c" (UID: "e21baeba-d058-495d-81b1-7a93f16372c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:55:04.050091 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:04.049955 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:04.050091 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:04.049969 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:04.050358 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:04.050105 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:04.050358 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:04.050212 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:05.050351 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:05.050311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:05.050917 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:05.050463 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:06.050273 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:06.050236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:06.050490 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:06.050290 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:06.050490 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:06.050383 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:06.050894 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:06.050524 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:07.050811 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:07.050771 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:07.051298 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:07.050923 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:08.049866 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:08.049826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:08.050087 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:08.049826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:08.050087 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:08.049984 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:08.050087 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:08.050002 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:09.050655 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:09.050617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:09.051287 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:09.050768 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:10.049989 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.049630 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:10.050083 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.049663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:10.050210 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:10.050114 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:10.050256 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:10.050236 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:10.308688 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.308640 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qhwlv" event={"ID":"0c64883f-3ead-4052-bea8-c09675421988","Type":"ContainerStarted","Data":"ad904dbdee73042bab36edd1dc64553adc3e7918d9c622dd7efdcdd1ef4fe972"} Jul 16 08:55:10.310284 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.310245 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xtljz" event={"ID":"04273c03-411a-4deb-b522-061973f7d4a7","Type":"ContainerStarted","Data":"97b612ac6c924e17486f11c07eaf62dbd1e5c8f3add114977b4caf1566a0c0a8"} Jul 16 08:55:10.311749 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.311719 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" event={"ID":"bc6ee5a6-68ee-436a-99d0-46b4abd359fe","Type":"ContainerStarted","Data":"837313a19eafa4f50f0e1d302eeb2dfc10f87e50ef2fce43fd6fa545c701b3ff"} Jul 16 08:55:10.313500 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.313470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-njtsp" event={"ID":"180ba7da-0767-42a2-9f01-4492fff3b648","Type":"ContainerStarted","Data":"771526cadbeac8e5aad99766d3c8a070db08020a590fc3b758a6faceb97cdcb5"} Jul 16 08:55:10.316327 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.316305 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 08:55:10.316674 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.316650 2574 generic.go:356] "Generic (PLEG): container finished" podID="08a75a91-2b97-4b16-8951-8d91c167dea5" containerID="89a8203cdd3cb80a585f9e07323641e4e91741f1dd49697b8a82f7d79d1ef0d2" exitCode=1 Jul 16 08:55:10.316777 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.316729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerStarted","Data":"fd558717956dc750c0cccf88e0f3d8f2cce9a159e4b54df757dec032127c4e95"} Jul 16 08:55:10.316826 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.316815 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerStarted","Data":"ecd27f93bfcee7d11d3d6aac285c6e9d115521ef1318ff3441bd568c3e31c291"} Jul 16 08:55:10.316877 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.316867 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerStarted","Data":"5fd77b9fef757123571184f81e399ded6ecf75238e4105f409c40b8af3ebdd0a"} Jul 16 08:55:10.316990 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.316916 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerStarted","Data":"47bbe46cd582a5550991355c6e27c63f6e8cea927d69ac2a39777ef921deb21a"} Jul 16 08:55:10.316990 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.316959 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerDied","Data":"89a8203cdd3cb80a585f9e07323641e4e91741f1dd49697b8a82f7d79d1ef0d2"} Jul 16 08:55:10.317102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.317003 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerStarted","Data":"63b9f33f2062cc27ca7ad79a842e26ec2f10e16f2b0d2c08a6b454131ff0d131"} Jul 16 08:55:10.318700 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.318671 2574 generic.go:356] "Generic (PLEG): container finished" podID="9dac8b01-f667-4faa-801b-c69c9655c829" containerID="02a6a938a1f1a910a1b3666589f63352738f827026432a072af52a459250b58b" exitCode=0 Jul 16 08:55:10.318888 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.318766 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" event={"ID":"9dac8b01-f667-4faa-801b-c69c9655c829","Type":"ContainerDied","Data":"02a6a938a1f1a910a1b3666589f63352738f827026432a072af52a459250b58b"} Jul 16 08:55:10.320799 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.320768 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lpps8" event={"ID":"928dda30-aac8-42e9-a376-9eb125067780","Type":"ContainerStarted","Data":"5f71b68fe9ca4353bfb3502e2aa3fb346e5cb1ff953a5593a03804b3407db2f6"} Jul 16 08:55:10.322631 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.322605 2574 generic.go:356] "Generic (PLEG): container finished" podID="6462d2c44e021ea56ae54eb1c6c365ab" containerID="53ade317fd21859048ba98551250d68fd58ba8ea523b6d0479585781b06053fb" exitCode=0 Jul 16 08:55:10.322734 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.322692 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" event={"ID":"6462d2c44e021ea56ae54eb1c6c365ab","Type":"ContainerDied","Data":"53ade317fd21859048ba98551250d68fd58ba8ea523b6d0479585781b06053fb"} Jul 16 08:55:10.324665 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.324260 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6j42v" event={"ID":"c7b6480a-2e28-4325-b994-4a1bd5516125","Type":"ContainerStarted","Data":"16dc5cd081c3c5fd5537bae66a1cf6825f719b6442dbc03a4b306545b414310f"} Jul 16 08:55:10.326777 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.326661 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" event={"ID":"8dfbfc1bca38cfe8c6a6944c393e430e","Type":"ContainerStarted","Data":"7cff7d113e702ad1eb366b54e04c398dd1e46f36973e3f1b761fb7f1786cf0e3"} Jul 16 08:55:10.326918 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.326846 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qhwlv" podStartSLOduration=2.27844118 podStartE2EDuration="23.326828978s" totalImagesPullingTime="21.048387798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:10.325900915 +0000 UTC m=+23.700035209" watchObservedRunningTime="2026-07-16 08:55:10.326828978 +0000 UTC m=+23.700963271" Jul 16 08:55:10.348706 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.348639 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xtljz" podStartSLOduration=2.256863469 podStartE2EDuration="23.348614043s" totalImagesPullingTime="21.091750574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:10.34760723 +0000 UTC m=+23.721741520" watchObservedRunningTime="2026-07-16 08:55:10.348614043 +0000 UTC m=+23.722748337" Jul 16 08:55:10.382412 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.382300 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lpps8" podStartSLOduration=2.234713584 podStartE2EDuration="23.382280076s" totalImagesPullingTime="21.147566492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:10.381714856 +0000 UTC m=+23.755849158" watchObservedRunningTime="2026-07-16 08:55:10.382280076 +0000 UTC m=+23.756414366" Jul 16 08:55:10.428716 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.428648 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-njtsp" podStartSLOduration=2.330553713 podStartE2EDuration="23.428623293s" totalImagesPullingTime="21.09806958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:10.427571384 +0000 UTC m=+23.801705674" watchObservedRunningTime="2026-07-16 08:55:10.428623293 +0000 UTC m=+23.802757584" Jul 16 08:55:10.445685 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.445631 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6j42v" podStartSLOduration=2.356673984 podStartE2EDuration="23.445612393s" totalImagesPullingTime="21.088938409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:10.444989418 +0000 UTC m=+23.819123709" watchObservedRunningTime="2026-07-16 08:55:10.445612393 +0000 UTC m=+23.819746682" Jul 16 08:55:10.460810 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:10.460750 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-59.ec2.internal" podStartSLOduration=22.460725557 podStartE2EDuration="22.460725557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:10.460137228 +0000 UTC m=+23.834271519" watchObservedRunningTime="2026-07-16 08:55:10.460725557 +0000 UTC m=+23.834859869" Jul 16 08:55:11.050480 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:11.050442 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:11.050744 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:11.050599 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:11.203137 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:11.203099 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 08:55:11.331342 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:11.331248 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" event={"ID":"bc6ee5a6-68ee-436a-99d0-46b4abd359fe","Type":"ContainerStarted","Data":"6a01d1e8a31e489cd14a9d92699ba59fbbc866409fd0c11c1198b652a51d71b2"} Jul 16 08:55:11.333474 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:11.333427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" event={"ID":"6462d2c44e021ea56ae54eb1c6c365ab","Type":"ContainerStarted","Data":"cdeead90f78a04d6f133ded75962fdd687c58d251e1cb445ceb47c8e4451af5e"} Jul 16 08:55:11.335607 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:11.335570 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vwvrj" event={"ID":"bd03ff02-e36a-47a4-b9bb-16b91534c524","Type":"ContainerStarted","Data":"5cc57c55ff9e6360f6887dc15ddb69fd2cc7b8ee765043050a25639504fe8d6b"} Jul 16 08:55:11.351045 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:11.350979 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-59.ec2.internal" podStartSLOduration=23.350957511 podStartE2EDuration="23.350957511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:11.350510475 +0000 UTC m=+24.724644778" watchObservedRunningTime="2026-07-16 08:55:11.350957511 +0000 UTC m=+24.725091801" Jul 16 08:55:11.365050 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:11.364991 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vwvrj" podStartSLOduration=3.335354024 podStartE2EDuration="24.364973906s" totalImagesPullingTime="21.029619882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:11.364782351 +0000 UTC m=+24.738916641" watchObservedRunningTime="2026-07-16 08:55:11.364973906 +0000 UTC m=+24.739108195" Jul 16 08:55:12.024764 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:12.024647 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T08:55:11.203128308Z","UUID":"01a15ff2-3c0a-4719-be1b-3d3121290750","Handler":null,"Name":"","Endpoint":""} Jul 16 08:55:12.026921 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:12.026822 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 08:55:12.026921 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:12.026869 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 08:55:12.049624 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:12.049591 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:12.049841 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:12.049591 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:12.049841 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:12.049739 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:12.049841 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:12.049821 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:12.979797 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:12.979751 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:55:12.980649 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:12.980628 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:55:13.049699 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:13.049660 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:13.049890 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:13.049782 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:13.343668 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:13.343564 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" event={"ID":"bc6ee5a6-68ee-436a-99d0-46b4abd359fe","Type":"ContainerStarted","Data":"a566ef6b36acbb679779ff594b938a2dd6d52d2e4c5ab55c62b06e43d961a578"} Jul 16 08:55:13.354330 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:13.354302 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 08:55:13.354714 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:13.354682 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerStarted","Data":"bc69b54ee939a92f891b94b28d81da8e5eb7d7e267ed849406ebcb4b845abf80"} Jul 16 08:55:13.354960 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:13.354941 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:55:13.355502 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:13.355485 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-njtsp" Jul 16 08:55:13.381429 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:13.381364 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4lwf" podStartSLOduration=2.795384528 podStartE2EDuration="26.381345458s" totalImagesPullingTime="23.58596093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:13.364544506 +0000 UTC m=+26.738678797" watchObservedRunningTime="2026-07-16 08:55:13.381345458 +0000 UTC m=+26.755479748" Jul 16 08:55:14.049816 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:14.049599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:14.050358 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:14.049655 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:14.050358 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:14.049924 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:14.050358 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:14.050015 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:15.050783 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:15.050581 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:15.051581 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:15.050872 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:15.363722 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:15.363690 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 08:55:15.364065 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:15.364031 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerStarted","Data":"b54b0dadd6e74a3faa1385dcb82172acf99ee1c649f3f3368485a0953903c277"} Jul 16 08:55:15.364364 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:15.364339 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:55:15.364514 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:15.364423 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:55:15.364612 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:15.364594 2574 scope.go:117] "RemoveContainer" containerID="89a8203cdd3cb80a585f9e07323641e4e91741f1dd49697b8a82f7d79d1ef0d2" Jul 16 08:55:15.366006 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:15.365967 2574 generic.go:356] "Generic (PLEG): container finished" podID="9dac8b01-f667-4faa-801b-c69c9655c829" containerID="909fa31ad5cc2f87733ebb99a383f21c5f42dd661d1c344b005953fba1b5dc6c" exitCode=0 Jul 16 08:55:15.366203 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:15.366050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" event={"ID":"9dac8b01-f667-4faa-801b-c69c9655c829","Type":"ContainerDied","Data":"909fa31ad5cc2f87733ebb99a383f21c5f42dd661d1c344b005953fba1b5dc6c"} Jul 16 08:55:15.383185 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:15.383145 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:55:16.050541 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.050503 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:16.050541 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.050531 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:16.050867 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:16.050665 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:16.050867 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:16.050751 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:16.373310 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.373286 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 08:55:16.373734 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.373703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" event={"ID":"08a75a91-2b97-4b16-8951-8d91c167dea5","Type":"ContainerStarted","Data":"78d2e815d8d7a09efdc6c1e84970f2efd0cc25a210f183435b688890994656c6"} Jul 16 08:55:16.374025 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.373976 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:55:16.391051 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.391015 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:55:16.401434 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.401365 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" podStartSLOduration=8.203747904 podStartE2EDuration="29.40134828s" totalImagesPullingTime="21.197600376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:16.400093153 +0000 UTC m=+29.774227488" watchObservedRunningTime="2026-07-16 08:55:16.40134828 +0000 UTC m=+29.775482570" Jul 16 08:55:16.653427 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.653389 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kbb9c"] Jul 16 08:55:16.653656 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.653517 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:16.653656 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:16.653632 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:16.656468 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.656423 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pd2px"] Jul 16 08:55:16.656648 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.656573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:16.656737 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:16.656712 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:16.657228 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.657202 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7zblf"] Jul 16 08:55:16.657359 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:16.657306 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:16.657437 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:16.657414 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:17.378058 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:17.378021 2574 generic.go:356] "Generic (PLEG): container finished" podID="9dac8b01-f667-4faa-801b-c69c9655c829" containerID="183ae0f4343f69aa9ed683a96b9b363031858edf8ba2c2abaa2f5c6a0d82dc87" exitCode=0 Jul 16 08:55:17.378750 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:17.378113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" event={"ID":"9dac8b01-f667-4faa-801b-c69c9655c829","Type":"ContainerDied","Data":"183ae0f4343f69aa9ed683a96b9b363031858edf8ba2c2abaa2f5c6a0d82dc87"} Jul 16 08:55:18.049847 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:18.049819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:18.050044 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:18.049861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:18.050044 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:18.049861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:18.050044 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:18.049926 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:18.050044 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:18.050029 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:18.050218 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:18.050095 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:19.385661 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:19.385445 2574 generic.go:356] "Generic (PLEG): container finished" podID="9dac8b01-f667-4faa-801b-c69c9655c829" containerID="119494aad47f6d0b8f7c32a9e2199a632f14e59292d03c89f5bb12aa2d765cea" exitCode=0 Jul 16 08:55:19.385661 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:19.385517 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" event={"ID":"9dac8b01-f667-4faa-801b-c69c9655c829","Type":"ContainerDied","Data":"119494aad47f6d0b8f7c32a9e2199a632f14e59292d03c89f5bb12aa2d765cea"} Jul 16 08:55:19.716658 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:19.716614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:19.716881 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:19.716769 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 08:55:19.716881 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:19.716838 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret podName:491e030c-547f-4c64-be98-eb0c8313b800 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:51.716821501 +0000 UTC m=+65.090955768 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret") pod "global-pull-secret-syncer-pd2px" (UID: "491e030c-547f-4c64-be98-eb0c8313b800") : object "kube-system"/"original-pull-secret" not registered Jul 16 08:55:19.816949 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:19.816915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:19.817134 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:19.817034 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:55:19.817134 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:19.817088 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs podName:7689dfa7-1f38-485d-a5e5-1fbea1c74f57 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:51.817073932 +0000 UTC m=+65.191208201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs") pod "network-metrics-daemon-7zblf" (UID: "7689dfa7-1f38-485d-a5e5-1fbea1c74f57") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 08:55:19.917783 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:19.917744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:19.917975 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:19.917958 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 08:55:19.918067 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:19.917998 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 08:55:19.918067 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:19.918015 2574 projected.go:196] Error preparing data for projected volume kube-api-access-ghljh for pod openshift-network-diagnostics/network-check-target-kbb9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:55:19.918194 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:19.918086 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh podName:e21baeba-d058-495d-81b1-7a93f16372c3 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:51.918064196 +0000 UTC m=+65.292198475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghljh" (UniqueName: "kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh") pod "network-check-target-kbb9c" (UID: "e21baeba-d058-495d-81b1-7a93f16372c3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 08:55:20.049760 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:20.049656 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:20.049760 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:20.049685 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:20.049760 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:20.049703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:20.050046 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:20.049794 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:20.050046 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:20.049933 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:20.050142 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:20.050045 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:22.049914 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.049867 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:22.050566 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.049916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:22.050566 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.049916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:22.050566 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:22.050026 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kbb9c" podUID="e21baeba-d058-495d-81b1-7a93f16372c3" Jul 16 08:55:22.050566 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:22.050099 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pd2px" podUID="491e030c-547f-4c64-be98-eb0c8313b800" Jul 16 08:55:22.050566 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:22.050210 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7zblf" podUID="7689dfa7-1f38-485d-a5e5-1fbea1c74f57" Jul 16 08:55:22.523899 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.523863 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-59.ec2.internal" event="NodeReady" Jul 16 08:55:22.524110 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.524024 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 08:55:22.581072 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.581031 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qkrk4"] Jul 16 08:55:22.612519 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.612480 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j6rmb"] Jul 16 08:55:22.612712 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.612589 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.615970 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.615762 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wm6pc\"" Jul 16 08:55:22.617024 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.616993 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 08:55:22.617182 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.617057 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 08:55:22.626500 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.626465 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qkrk4"] Jul 16 08:55:22.626653 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.626529 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6rmb"] Jul 16 08:55:22.626653 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.626611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6rmb" Jul 16 08:55:22.629455 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.629202 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 08:55:22.629455 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.629267 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8tdqp\"" Jul 16 08:55:22.629455 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.629202 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 08:55:22.629455 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.629363 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 08:55:22.737927 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.737890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837b276c-808f-46c4-88c2-6c590b5d6c02-tmp-dir\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.737927 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.737933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxpqj\" (UniqueName: \"kubernetes.io/projected/5abfceca-c1c1-42a0-bc57-ff208b0b63fd-kube-api-access-gxpqj\") pod \"ingress-canary-j6rmb\" (UID: \"5abfceca-c1c1-42a0-bc57-ff208b0b63fd\") " pod="openshift-ingress-canary/ingress-canary-j6rmb" Jul 16 08:55:22.738201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.737970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/837b276c-808f-46c4-88c2-6c590b5d6c02-config-volume\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.738201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.738022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5abfceca-c1c1-42a0-bc57-ff208b0b63fd-cert\") pod \"ingress-canary-j6rmb\" (UID: \"5abfceca-c1c1-42a0-bc57-ff208b0b63fd\") " pod="openshift-ingress-canary/ingress-canary-j6rmb" Jul 16 08:55:22.738201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.738128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpmg7\" (UniqueName: \"kubernetes.io/projected/837b276c-808f-46c4-88c2-6c590b5d6c02-kube-api-access-vpmg7\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.738388 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.738267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.839338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.839246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/837b276c-808f-46c4-88c2-6c590b5d6c02-config-volume\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.839338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.839289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5abfceca-c1c1-42a0-bc57-ff208b0b63fd-cert\") pod \"ingress-canary-j6rmb\" (UID: \"5abfceca-c1c1-42a0-bc57-ff208b0b63fd\") " pod="openshift-ingress-canary/ingress-canary-j6rmb" Jul 16 08:55:22.839338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.839340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpmg7\" (UniqueName: \"kubernetes.io/projected/837b276c-808f-46c4-88c2-6c590b5d6c02-kube-api-access-vpmg7\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.839623 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.839405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.839623 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.839429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837b276c-808f-46c4-88c2-6c590b5d6c02-tmp-dir\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.839623 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.839449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxpqj\" (UniqueName: \"kubernetes.io/projected/5abfceca-c1c1-42a0-bc57-ff208b0b63fd-kube-api-access-gxpqj\") pod \"ingress-canary-j6rmb\" (UID: \"5abfceca-c1c1-42a0-bc57-ff208b0b63fd\") " pod="openshift-ingress-canary/ingress-canary-j6rmb" Jul 16 08:55:22.839773 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:22.839635 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 08:55:22.839773 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:22.839722 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls podName:837b276c-808f-46c4-88c2-6c590b5d6c02 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:23.33969732 +0000 UTC m=+36.713831596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls") pod "dns-default-qkrk4" (UID: "837b276c-808f-46c4-88c2-6c590b5d6c02") : secret "dns-default-metrics-tls" not found Jul 16 08:55:22.839873 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.839800 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/837b276c-808f-46c4-88c2-6c590b5d6c02-tmp-dir\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.840014 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.839991 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/837b276c-808f-46c4-88c2-6c590b5d6c02-config-volume\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.844589 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.844559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5abfceca-c1c1-42a0-bc57-ff208b0b63fd-cert\") pod \"ingress-canary-j6rmb\" (UID: \"5abfceca-c1c1-42a0-bc57-ff208b0b63fd\") " pod="openshift-ingress-canary/ingress-canary-j6rmb" Jul 16 08:55:22.850972 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.850942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpmg7\" (UniqueName: \"kubernetes.io/projected/837b276c-808f-46c4-88c2-6c590b5d6c02-kube-api-access-vpmg7\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:22.851142 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.851126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxpqj\" (UniqueName: \"kubernetes.io/projected/5abfceca-c1c1-42a0-bc57-ff208b0b63fd-kube-api-access-gxpqj\") pod \"ingress-canary-j6rmb\" (UID: \"5abfceca-c1c1-42a0-bc57-ff208b0b63fd\") " pod="openshift-ingress-canary/ingress-canary-j6rmb" Jul 16 08:55:22.937437 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:22.937393 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j6rmb" Jul 16 08:55:23.343462 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:23.343420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:23.344357 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:23.343574 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 08:55:23.344357 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:23.343644 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls podName:837b276c-808f-46c4-88c2-6c590b5d6c02 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:24.343627946 +0000 UTC m=+37.717762214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls") pod "dns-default-qkrk4" (UID: "837b276c-808f-46c4-88c2-6c590b5d6c02") : secret "dns-default-metrics-tls" not found Jul 16 08:55:23.621254 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:23.621024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j6rmb"] Jul 16 08:55:23.709981 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:55:23.709947 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abfceca_c1c1_42a0_bc57_ff208b0b63fd.slice/crio-951062ab50b3acd01f51de114c6e5c347604453aaab02d26651538a4fa73841c WatchSource:0}: Error finding container 951062ab50b3acd01f51de114c6e5c347604453aaab02d26651538a4fa73841c: Status 404 returned error can't find the container with id 951062ab50b3acd01f51de114c6e5c347604453aaab02d26651538a4fa73841c Jul 16 08:55:24.049662 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.049613 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:24.049662 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.049654 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:24.049929 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.049658 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:24.052393 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.052364 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 08:55:24.052393 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.052367 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7ks4\"" Jul 16 08:55:24.052607 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.052413 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 08:55:24.052607 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.052474 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rgclf\"" Jul 16 08:55:24.053347 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.053331 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 08:55:24.053347 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.053347 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 08:55:24.352928 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.352824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:24.353347 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:24.353015 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 08:55:24.353347 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:24.353097 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls podName:837b276c-808f-46c4-88c2-6c590b5d6c02 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:26.353079415 +0000 UTC m=+39.727213686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls") pod "dns-default-qkrk4" (UID: "837b276c-808f-46c4-88c2-6c590b5d6c02") : secret "dns-default-metrics-tls" not found Jul 16 08:55:24.401856 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.401813 2574 generic.go:356] "Generic (PLEG): container finished" podID="9dac8b01-f667-4faa-801b-c69c9655c829" containerID="ee508ad13ef1fea5fe771852fe0508e3aeebe39720f343c77a4d9bb98ead4d84" exitCode=0 Jul 16 08:55:24.402064 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.401901 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" event={"ID":"9dac8b01-f667-4faa-801b-c69c9655c829","Type":"ContainerDied","Data":"ee508ad13ef1fea5fe771852fe0508e3aeebe39720f343c77a4d9bb98ead4d84"} Jul 16 08:55:24.403131 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:24.403105 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6rmb" event={"ID":"5abfceca-c1c1-42a0-bc57-ff208b0b63fd","Type":"ContainerStarted","Data":"951062ab50b3acd01f51de114c6e5c347604453aaab02d26651538a4fa73841c"} Jul 16 08:55:25.409883 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:25.409831 2574 generic.go:356] "Generic (PLEG): container finished" podID="9dac8b01-f667-4faa-801b-c69c9655c829" containerID="6f8b9b5502ef0f573e6cde0b8893de3cfe5da3b25fa413627cf28ff25b4d8a9d" exitCode=0 Jul 16 08:55:25.410450 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:25.409888 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" event={"ID":"9dac8b01-f667-4faa-801b-c69c9655c829","Type":"ContainerDied","Data":"6f8b9b5502ef0f573e6cde0b8893de3cfe5da3b25fa413627cf28ff25b4d8a9d"} Jul 16 08:55:26.369503 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:26.369298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:26.369683 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:26.369455 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 08:55:26.369683 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:26.369611 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls podName:837b276c-808f-46c4-88c2-6c590b5d6c02 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:30.369588578 +0000 UTC m=+43.743722861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls") pod "dns-default-qkrk4" (UID: "837b276c-808f-46c4-88c2-6c590b5d6c02") : secret "dns-default-metrics-tls" not found Jul 16 08:55:26.415711 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:26.415667 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" event={"ID":"9dac8b01-f667-4faa-801b-c69c9655c829","Type":"ContainerStarted","Data":"92e4550aa401054cbe3fcfd1d5c7f21e96993a63be8a78f57feb3b762796bb9b"} Jul 16 08:55:26.417249 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:26.417144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j6rmb" event={"ID":"5abfceca-c1c1-42a0-bc57-ff208b0b63fd","Type":"ContainerStarted","Data":"d0acfb1b482758c16de76cffd9a2808b45c74824efebfdf5b04965357858287f"} Jul 16 08:55:26.439582 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:26.439520 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2zqbr" podStartSLOduration=7.376613289 podStartE2EDuration="39.43949761s" totalImagesPullingTime="32.062884321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:26.43761774 +0000 UTC m=+39.811752029" watchObservedRunningTime="2026-07-16 08:55:26.43949761 +0000 UTC m=+39.813631901" Jul 16 08:55:26.453742 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:26.453622 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j6rmb" podStartSLOduration=2.304528928 podStartE2EDuration="4.453599152s" totalImagesPullingTime="2.149070224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:26.453284394 +0000 UTC m=+39.827418684" watchObservedRunningTime="2026-07-16 08:55:26.453599152 +0000 UTC m=+39.827733439" Jul 16 08:55:30.397695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:30.397649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:30.398292 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:30.397780 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 08:55:30.398292 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:30.397835 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls podName:837b276c-808f-46c4-88c2-6c590b5d6c02 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:38.397820539 +0000 UTC m=+51.771954808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls") pod "dns-default-qkrk4" (UID: "837b276c-808f-46c4-88c2-6c590b5d6c02") : secret "dns-default-metrics-tls" not found Jul 16 08:55:38.454860 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:38.454817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:38.455379 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:38.454973 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 08:55:38.458828 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:38.458778 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls podName:837b276c-808f-46c4-88c2-6c590b5d6c02 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:54.458739169 +0000 UTC m=+67.832873440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls") pod "dns-default-qkrk4" (UID: "837b276c-808f-46c4-88c2-6c590b5d6c02") : secret "dns-default-metrics-tls" not found Jul 16 08:55:46.684691 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.684656 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5"] Jul 16 08:55:46.687528 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.687507 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.689789 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.689758 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 16 08:55:46.689991 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.689974 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 16 08:55:46.690084 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.690028 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 16 08:55:46.690084 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.690068 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 16 08:55:46.690916 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.690896 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-njkf5\"" Jul 16 08:55:46.695986 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.695943 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5"] Jul 16 08:55:46.782521 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.782485 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9"] Jul 16 08:55:46.784602 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.784585 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:46.788229 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.788205 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 16 08:55:46.788641 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.788617 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 08:55:46.788775 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.788657 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qlq58\"" Jul 16 08:55:46.788896 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.788880 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 08:55:46.788975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.788890 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 16 08:55:46.789564 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.789543 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bd4d58779-5qp9b"] Jul 16 08:55:46.792309 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.792289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:46.794754 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.794732 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 08:55:46.794875 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.794808 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zggx9\"" Jul 16 08:55:46.795301 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.795283 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 08:55:46.795824 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.795806 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 08:55:46.797956 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.797876 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9"] Jul 16 08:55:46.800845 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.800821 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 08:55:46.805439 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.805414 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bd4d58779-5qp9b"] Jul 16 08:55:46.813285 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.813258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f001e84b-5fab-4319-814e-19e40d8cc591-config\") pod \"service-ca-operator-7695fc6c8c-x9ch5\" (UID: \"f001e84b-5fab-4319-814e-19e40d8cc591\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.813432 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.813290 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ssvx\" (UniqueName: \"kubernetes.io/projected/f001e84b-5fab-4319-814e-19e40d8cc591-kube-api-access-4ssvx\") pod \"service-ca-operator-7695fc6c8c-x9ch5\" (UID: \"f001e84b-5fab-4319-814e-19e40d8cc591\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.813432 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.813364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f001e84b-5fab-4319-814e-19e40d8cc591-serving-cert\") pod \"service-ca-operator-7695fc6c8c-x9ch5\" (UID: \"f001e84b-5fab-4319-814e-19e40d8cc591\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.874971 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.874935 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z"] Jul 16 08:55:46.877286 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.877269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:46.879770 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.879742 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 16 08:55:46.879903 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.879792 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 16 08:55:46.879963 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.879906 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 16 08:55:46.880046 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.880014 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 16 08:55:46.880046 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.880039 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9qszd\"" Jul 16 08:55:46.881521 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.881499 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-8g4xv"] Jul 16 08:55:46.884002 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.883768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:46.886189 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.886168 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 16 08:55:46.886285 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.886191 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zpbqb\"" Jul 16 08:55:46.886285 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.886261 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 08:55:46.886285 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.886261 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 08:55:46.886471 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.886337 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 16 08:55:46.893042 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.893019 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 16 08:55:46.897389 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.897363 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z"] Jul 16 08:55:46.899707 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.899681 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-8g4xv"] Jul 16 08:55:46.914170 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-registry-certificates\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:46.914369 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/936e4c26-9180-4276-ab88-95b85db3653b-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:46.914369 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhmn\" (UniqueName: \"kubernetes.io/projected/936e4c26-9180-4276-ab88-95b85db3653b-kube-api-access-dxhmn\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:46.914369 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f001e84b-5fab-4319-814e-19e40d8cc591-config\") pod \"service-ca-operator-7695fc6c8c-x9ch5\" (UID: \"f001e84b-5fab-4319-814e-19e40d8cc591\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.914369 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ssvx\" (UniqueName: \"kubernetes.io/projected/f001e84b-5fab-4319-814e-19e40d8cc591-kube-api-access-4ssvx\") pod \"service-ca-operator-7695fc6c8c-x9ch5\" (UID: \"f001e84b-5fab-4319-814e-19e40d8cc591\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.914369 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4vw9\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-kube-api-access-k4vw9\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:46.914622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914379 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:46.914622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-trusted-ca\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:46.914622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f001e84b-5fab-4319-814e-19e40d8cc591-serving-cert\") pod \"service-ca-operator-7695fc6c8c-x9ch5\" (UID: \"f001e84b-5fab-4319-814e-19e40d8cc591\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.914622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-image-registry-private-configuration\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:46.914622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-bound-sa-token\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:46.914622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:46.914622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662eb4c1-fee9-43f7-a560-09179fc18194-ca-trust-extracted\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:46.914949 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-installation-pull-secrets\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:46.915020 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.914996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f001e84b-5fab-4319-814e-19e40d8cc591-config\") pod \"service-ca-operator-7695fc6c8c-x9ch5\" (UID: \"f001e84b-5fab-4319-814e-19e40d8cc591\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.918512 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.918478 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f001e84b-5fab-4319-814e-19e40d8cc591-serving-cert\") pod \"service-ca-operator-7695fc6c8c-x9ch5\" (UID: \"f001e84b-5fab-4319-814e-19e40d8cc591\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.930907 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.930868 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ssvx\" (UniqueName: \"kubernetes.io/projected/f001e84b-5fab-4319-814e-19e40d8cc591-kube-api-access-4ssvx\") pod \"service-ca-operator-7695fc6c8c-x9ch5\" (UID: \"f001e84b-5fab-4319-814e-19e40d8cc591\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:46.998938 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:46.998849 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-njkf5\"" Jul 16 08:55:47.006862 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.006833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" Jul 16 08:55:47.015642 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.015612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/019c8aea-66ce-4497-bc6a-550bdefad29c-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.015779 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.015650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/019c8aea-66ce-4497-bc6a-550bdefad29c-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.015833 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.015782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4526fa9c-78bc-463d-8244-cf63b0c783bb-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-j4c9z\" (UID: \"4526fa9c-78bc-463d-8244-cf63b0c783bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.015892 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.015826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhmn\" (UniqueName: \"kubernetes.io/projected/936e4c26-9180-4276-ab88-95b85db3653b-kube-api-access-dxhmn\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:47.015946 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.015884 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.015946 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.015920 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-bound-sa-token\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.016045 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.015948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:47.016045 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.015978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-installation-pull-secrets\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.016045 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnnq\" (UniqueName: \"kubernetes.io/projected/019c8aea-66ce-4497-bc6a-550bdefad29c-kube-api-access-jlnnq\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.016210 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-trusted-ca\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.016210 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8kq\" (UniqueName: \"kubernetes.io/projected/4526fa9c-78bc-463d-8244-cf63b0c783bb-kube-api-access-tz8kq\") pod \"kube-storage-version-migrator-operator-799cb7f958-j4c9z\" (UID: \"4526fa9c-78bc-463d-8244-cf63b0c783bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.016210 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-registry-certificates\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.016210 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/019c8aea-66ce-4497-bc6a-550bdefad29c-serving-cert\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.016396 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/936e4c26-9180-4276-ab88-95b85db3653b-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:47.016396 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4vw9\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-kube-api-access-k4vw9\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.016510 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4526fa9c-78bc-463d-8244-cf63b0c783bb-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-j4c9z\" (UID: \"4526fa9c-78bc-463d-8244-cf63b0c783bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.016510 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-image-registry-private-configuration\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.016617 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/019c8aea-66ce-4497-bc6a-550bdefad29c-snapshots\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.016617 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662eb4c1-fee9-43f7-a560-09179fc18194-ca-trust-extracted\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.016617 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/019c8aea-66ce-4497-bc6a-550bdefad29c-tmp\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.016751 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-registry-certificates\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.016872 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.016847 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662eb4c1-fee9-43f7-a560-09179fc18194-ca-trust-extracted\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.018925 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.018898 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 08:55:47.019093 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.019072 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 16 08:55:47.019188 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.019105 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 08:55:47.019252 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.019233 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 08:55:47.019828 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.019808 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 16 08:55:47.024215 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.024169 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 08:55:47.026095 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.026073 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 08:55:47.026095 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.026093 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd4d58779-5qp9b: secret "image-registry-tls" not found Jul 16 08:55:47.026350 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.026138 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:47.026350 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.026176 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls podName:662eb4c1-fee9-43f7-a560-09179fc18194 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:47.526133523 +0000 UTC m=+60.900267795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls") pod "image-registry-bd4d58779-5qp9b" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194") : secret "image-registry-tls" not found Jul 16 08:55:47.026350 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.026224 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls podName:936e4c26-9180-4276-ab88-95b85db3653b nodeName:}" failed. No retries permitted until 2026-07-16 08:55:47.526206965 +0000 UTC m=+60.900341236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-54bw9" (UID: "936e4c26-9180-4276-ab88-95b85db3653b") : secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:47.026519 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.026500 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 08:55:47.027280 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.027248 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-bound-sa-token\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.027479 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.027454 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-trusted-ca\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.027781 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.027754 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4vw9\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-kube-api-access-k4vw9\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.027871 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.027790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/936e4c26-9180-4276-ab88-95b85db3653b-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:47.029742 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.029717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-installation-pull-secrets\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.029838 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.029762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-image-registry-private-configuration\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.036801 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.036771 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 08:55:47.047687 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.047660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhmn\" (UniqueName: \"kubernetes.io/projected/936e4c26-9180-4276-ab88-95b85db3653b-kube-api-access-dxhmn\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:47.117009 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.116979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnnq\" (UniqueName: \"kubernetes.io/projected/019c8aea-66ce-4497-bc6a-550bdefad29c-kube-api-access-jlnnq\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.117235 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117017 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8kq\" (UniqueName: \"kubernetes.io/projected/4526fa9c-78bc-463d-8244-cf63b0c783bb-kube-api-access-tz8kq\") pod \"kube-storage-version-migrator-operator-799cb7f958-j4c9z\" (UID: \"4526fa9c-78bc-463d-8244-cf63b0c783bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.117235 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/019c8aea-66ce-4497-bc6a-550bdefad29c-serving-cert\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.117235 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4526fa9c-78bc-463d-8244-cf63b0c783bb-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-j4c9z\" (UID: \"4526fa9c-78bc-463d-8244-cf63b0c783bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.117235 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/019c8aea-66ce-4497-bc6a-550bdefad29c-snapshots\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.117458 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/019c8aea-66ce-4497-bc6a-550bdefad29c-tmp\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.117458 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/019c8aea-66ce-4497-bc6a-550bdefad29c-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.117458 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/019c8aea-66ce-4497-bc6a-550bdefad29c-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.117458 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4526fa9c-78bc-463d-8244-cf63b0c783bb-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-j4c9z\" (UID: \"4526fa9c-78bc-463d-8244-cf63b0c783bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.117858 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/019c8aea-66ce-4497-bc6a-550bdefad29c-tmp\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.117933 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.117904 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/019c8aea-66ce-4497-bc6a-550bdefad29c-snapshots\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.119581 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.119562 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 16 08:55:47.119669 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.119604 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 16 08:55:47.119807 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.119794 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 16 08:55:47.119882 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.119866 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 16 08:55:47.123774 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.123752 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 16 08:55:47.127965 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.127937 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/019c8aea-66ce-4497-bc6a-550bdefad29c-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.128296 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.128270 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4526fa9c-78bc-463d-8244-cf63b0c783bb-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-j4c9z\" (UID: \"4526fa9c-78bc-463d-8244-cf63b0c783bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.128676 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.128654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/019c8aea-66ce-4497-bc6a-550bdefad29c-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.129221 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.129175 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 08:55:47.129221 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.129195 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 16 08:55:47.129979 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.129954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4526fa9c-78bc-463d-8244-cf63b0c783bb-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-j4c9z\" (UID: \"4526fa9c-78bc-463d-8244-cf63b0c783bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.130201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.130183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/019c8aea-66ce-4497-bc6a-550bdefad29c-serving-cert\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.139327 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.139302 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 16 08:55:47.139327 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.139302 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 08:55:47.149821 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.149790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8kq\" (UniqueName: \"kubernetes.io/projected/4526fa9c-78bc-463d-8244-cf63b0c783bb-kube-api-access-tz8kq\") pod \"kube-storage-version-migrator-operator-799cb7f958-j4c9z\" (UID: \"4526fa9c-78bc-463d-8244-cf63b0c783bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.149977 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.149849 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnnq\" (UniqueName: \"kubernetes.io/projected/019c8aea-66ce-4497-bc6a-550bdefad29c-kube-api-access-jlnnq\") pod \"insights-operator-68d9dd94cf-8g4xv\" (UID: \"019c8aea-66ce-4497-bc6a-550bdefad29c\") " pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.155756 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.155724 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5"] Jul 16 08:55:47.158951 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:55:47.158926 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf001e84b_5fab_4319_814e_19e40d8cc591.slice/crio-59f67a464326652cea93fb510d5a146afef5f50e69037415d824299a60132ead WatchSource:0}: Error finding container 59f67a464326652cea93fb510d5a146afef5f50e69037415d824299a60132ead: Status 404 returned error can't find the container with id 59f67a464326652cea93fb510d5a146afef5f50e69037415d824299a60132ead Jul 16 08:55:47.190420 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.190390 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9qszd\"" Jul 16 08:55:47.197416 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.197388 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zpbqb\"" Jul 16 08:55:47.198397 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.198378 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" Jul 16 08:55:47.206126 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.206100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" Jul 16 08:55:47.365028 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.364998 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z"] Jul 16 08:55:47.367731 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:55:47.367703 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4526fa9c_78bc_463d_8244_cf63b0c783bb.slice/crio-1a6a76f9311e65e53cb8b8b580806b252ea6f9b09f24d8dc5e23428fda4353fe WatchSource:0}: Error finding container 1a6a76f9311e65e53cb8b8b580806b252ea6f9b09f24d8dc5e23428fda4353fe: Status 404 returned error can't find the container with id 1a6a76f9311e65e53cb8b8b580806b252ea6f9b09f24d8dc5e23428fda4353fe Jul 16 08:55:47.375852 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.375826 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-8g4xv"] Jul 16 08:55:47.379610 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:55:47.379581 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019c8aea_66ce_4497_bc6a_550bdefad29c.slice/crio-ea6634f135f511ee9d8c07d26c690b284887a82e70675e4f2c109ab9b4a6dee9 WatchSource:0}: Error finding container ea6634f135f511ee9d8c07d26c690b284887a82e70675e4f2c109ab9b4a6dee9: Status 404 returned error can't find the container with id ea6634f135f511ee9d8c07d26c690b284887a82e70675e4f2c109ab9b4a6dee9 Jul 16 08:55:47.467905 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.467867 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" event={"ID":"f001e84b-5fab-4319-814e-19e40d8cc591","Type":"ContainerStarted","Data":"59f67a464326652cea93fb510d5a146afef5f50e69037415d824299a60132ead"} Jul 16 08:55:47.468982 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.468959 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" event={"ID":"019c8aea-66ce-4497-bc6a-550bdefad29c","Type":"ContainerStarted","Data":"ea6634f135f511ee9d8c07d26c690b284887a82e70675e4f2c109ab9b4a6dee9"} Jul 16 08:55:47.470029 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.469991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" event={"ID":"4526fa9c-78bc-463d-8244-cf63b0c783bb","Type":"ContainerStarted","Data":"1a6a76f9311e65e53cb8b8b580806b252ea6f9b09f24d8dc5e23428fda4353fe"} Jul 16 08:55:47.621982 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.621944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:47.621982 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:47.621984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:47.622270 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.622088 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:47.622270 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.622111 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 08:55:47.622270 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.622134 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd4d58779-5qp9b: secret "image-registry-tls" not found Jul 16 08:55:47.622270 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.622172 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls podName:936e4c26-9180-4276-ab88-95b85db3653b nodeName:}" failed. No retries permitted until 2026-07-16 08:55:48.622130878 +0000 UTC m=+61.996265147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-54bw9" (UID: "936e4c26-9180-4276-ab88-95b85db3653b") : secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:47.622270 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:47.622202 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls podName:662eb4c1-fee9-43f7-a560-09179fc18194 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:48.622184926 +0000 UTC m=+61.996319197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls") pod "image-registry-bd4d58779-5qp9b" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194") : secret "image-registry-tls" not found Jul 16 08:55:48.393965 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:48.393825 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lf9r4" Jul 16 08:55:48.632004 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:48.631959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:48.632269 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:48.632027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:48.632269 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:48.632214 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:48.632396 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:48.632302 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls podName:936e4c26-9180-4276-ab88-95b85db3653b nodeName:}" failed. No retries permitted until 2026-07-16 08:55:50.632279839 +0000 UTC m=+64.006414132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-54bw9" (UID: "936e4c26-9180-4276-ab88-95b85db3653b") : secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:48.632396 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:48.632219 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 08:55:48.632396 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:48.632329 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd4d58779-5qp9b: secret "image-registry-tls" not found Jul 16 08:55:48.632521 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:48.632402 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls podName:662eb4c1-fee9-43f7-a560-09179fc18194 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:50.632380577 +0000 UTC m=+64.006514845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls") pod "image-registry-bd4d58779-5qp9b" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194") : secret "image-registry-tls" not found Jul 16 08:55:50.480253 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:50.480195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" event={"ID":"f001e84b-5fab-4319-814e-19e40d8cc591","Type":"ContainerStarted","Data":"5ffea6d052e1aa22ef23f9170b3cbec0ac7cfe748ae318577063c41768294ba1"} Jul 16 08:55:50.482782 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:50.482707 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" event={"ID":"019c8aea-66ce-4497-bc6a-550bdefad29c","Type":"ContainerStarted","Data":"d9b17b02454fa564fa62426b58dd59144d1b33c6ef139aa32d2b9a995133cfa8"} Jul 16 08:55:50.484655 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:50.484613 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" event={"ID":"4526fa9c-78bc-463d-8244-cf63b0c783bb","Type":"ContainerStarted","Data":"9987df9d6ac41f1f6d753d8f8b81b47e44700f2db58c96ebd56109b0006f3f36"} Jul 16 08:55:50.505059 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:50.504990 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" podStartSLOduration=1.480398066 podStartE2EDuration="4.504968256s" totalImagesPullingTime="3.02457019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:50.503429877 +0000 UTC m=+63.877564168" watchObservedRunningTime="2026-07-16 08:55:50.504968256 +0000 UTC m=+63.879102545" Jul 16 08:55:50.645231 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:50.645126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:50.645231 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:50.645195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:50.645606 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:50.645304 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:50.645606 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:50.645392 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls podName:936e4c26-9180-4276-ab88-95b85db3653b nodeName:}" failed. No retries permitted until 2026-07-16 08:55:54.645370324 +0000 UTC m=+68.019504599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-54bw9" (UID: "936e4c26-9180-4276-ab88-95b85db3653b") : secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:50.645606 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:50.645310 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 08:55:50.645606 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:50.645422 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd4d58779-5qp9b: secret "image-registry-tls" not found Jul 16 08:55:50.645606 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:50.645482 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls podName:662eb4c1-fee9-43f7-a560-09179fc18194 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:54.645465026 +0000 UTC m=+68.019599301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls") pod "image-registry-bd4d58779-5qp9b" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194") : secret "image-registry-tls" not found Jul 16 08:55:51.503863 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.503775 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" podStartSLOduration=2.2549311149999998 podStartE2EDuration="5.503752703s" totalImagesPullingTime="3.248821588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:51.50306027 +0000 UTC m=+64.877194561" watchObservedRunningTime="2026-07-16 08:55:51.503752703 +0000 UTC m=+64.877886994" Jul 16 08:55:51.504431 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.504112 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" podStartSLOduration=2.461992305 podStartE2EDuration="5.504098845s" totalImagesPullingTime="3.04210654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:50.524607777 +0000 UTC m=+63.898742068" watchObservedRunningTime="2026-07-16 08:55:51.504098845 +0000 UTC m=+64.878233127" Jul 16 08:55:51.755626 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.755529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:51.759434 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.759406 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 08:55:51.768583 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.768547 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/491e030c-547f-4c64-be98-eb0c8313b800-original-pull-secret\") pod \"global-pull-secret-syncer-pd2px\" (UID: \"491e030c-547f-4c64-be98-eb0c8313b800\") " pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:51.856103 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.856064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:55:51.859250 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.859211 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 08:55:51.866632 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:51.866596 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 08:55:51.866880 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:51.866690 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs podName:7689dfa7-1f38-485d-a5e5-1fbea1c74f57 nodeName:}" failed. No retries permitted until 2026-07-16 08:56:55.866667654 +0000 UTC m=+129.240801928 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs") pod "network-metrics-daemon-7zblf" (UID: "7689dfa7-1f38-485d-a5e5-1fbea1c74f57") : secret "metrics-daemon-secret" not found Jul 16 08:55:51.875148 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.875111 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bmcmn"] Jul 16 08:55:51.880979 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.880952 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:51.883304 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.883280 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 08:55:51.883441 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.883283 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 08:55:51.883441 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.883333 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b8kz8\"" Jul 16 08:55:51.897772 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.897741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmcmn"] Jul 16 08:55:51.957507 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.957459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90534626-e36c-448a-93f8-159f941e95b8-data-volume\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:51.957507 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.957508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:51.957752 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.957623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90534626-e36c-448a-93f8-159f941e95b8-crio-socket\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:51.957752 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.957675 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x94q\" (UniqueName: \"kubernetes.io/projected/90534626-e36c-448a-93f8-159f941e95b8-kube-api-access-7x94q\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:51.957752 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.957730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:51.957864 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.957813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:51.961798 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.961764 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 08:55:51.971253 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.971218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pd2px" Jul 16 08:55:51.971454 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.971420 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 08:55:51.982855 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:51.982826 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghljh\" (UniqueName: \"kubernetes.io/projected/e21baeba-d058-495d-81b1-7a93f16372c3-kube-api-access-ghljh\") pod \"network-check-target-kbb9c\" (UID: \"e21baeba-d058-495d-81b1-7a93f16372c3\") " pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:52.058888 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.058852 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.059049 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:52.059028 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 08:55:52.059102 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.059032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90534626-e36c-448a-93f8-159f941e95b8-data-volume\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.059170 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:52.059094 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:52.559077233 +0000 UTC m=+65.933211505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : secret "insights-runtime-extractor-tls" not found Jul 16 08:55:52.059246 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.059171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.059246 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:52.059225 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 08:55:52.059340 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.059245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90534626-e36c-448a-93f8-159f941e95b8-crio-socket\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.059340 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.059287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x94q\" (UniqueName: \"kubernetes.io/projected/90534626-e36c-448a-93f8-159f941e95b8-kube-api-access-7x94q\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.059340 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:52.059295 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:52.559280926 +0000 UTC m=+65.933415205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : configmap "kube-rbac-proxy" not found Jul 16 08:55:52.059477 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.059372 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90534626-e36c-448a-93f8-159f941e95b8-crio-socket\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.059477 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.059394 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90534626-e36c-448a-93f8-159f941e95b8-data-volume\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.082540 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.082510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x94q\" (UniqueName: \"kubernetes.io/projected/90534626-e36c-448a-93f8-159f941e95b8-kube-api-access-7x94q\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.105586 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.105550 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pd2px"] Jul 16 08:55:52.108725 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:55:52.108698 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491e030c_547f_4c64_be98_eb0c8313b800.slice/crio-08d36dd1fc1c38eff70a271c1518f9bd9ed079e5b81cfbae900a5f811e7ea7bd WatchSource:0}: Error finding container 08d36dd1fc1c38eff70a271c1518f9bd9ed079e5b81cfbae900a5f811e7ea7bd: Status 404 returned error can't find the container with id 08d36dd1fc1c38eff70a271c1518f9bd9ed079e5b81cfbae900a5f811e7ea7bd Jul 16 08:55:52.262572 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.262537 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rgclf\"" Jul 16 08:55:52.270729 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.270645 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:52.407774 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.407739 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kbb9c"] Jul 16 08:55:52.411257 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:55:52.411227 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21baeba_d058_495d_81b1_7a93f16372c3.slice/crio-8f928ff110bccfde5224fff3332e8919c2eccaca1ac647529d5dfdd8c3178c16 WatchSource:0}: Error finding container 8f928ff110bccfde5224fff3332e8919c2eccaca1ac647529d5dfdd8c3178c16: Status 404 returned error can't find the container with id 8f928ff110bccfde5224fff3332e8919c2eccaca1ac647529d5dfdd8c3178c16 Jul 16 08:55:52.491376 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.491341 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pd2px" event={"ID":"491e030c-547f-4c64-be98-eb0c8313b800","Type":"ContainerStarted","Data":"08d36dd1fc1c38eff70a271c1518f9bd9ed079e5b81cfbae900a5f811e7ea7bd"} Jul 16 08:55:52.492437 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.492408 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kbb9c" event={"ID":"e21baeba-d058-495d-81b1-7a93f16372c3","Type":"ContainerStarted","Data":"8f928ff110bccfde5224fff3332e8919c2eccaca1ac647529d5dfdd8c3178c16"} Jul 16 08:55:52.564923 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.564677 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.564923 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:52.564765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:52.564923 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:52.564781 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 08:55:52.564923 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:52.564819 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 08:55:52.564923 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:52.564847 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:53.564828029 +0000 UTC m=+66.938962318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : secret "insights-runtime-extractor-tls" not found Jul 16 08:55:52.564923 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:52.564865 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:53.564856883 +0000 UTC m=+66.938991155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : configmap "kube-rbac-proxy" not found Jul 16 08:55:53.572923 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:53.572881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:53.573448 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:53.573036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:53.573448 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:53.573110 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 08:55:53.573448 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:53.573134 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 08:55:53.573448 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:53.573212 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:55.573189578 +0000 UTC m=+68.947323848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : secret "insights-runtime-extractor-tls" not found Jul 16 08:55:53.573448 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:53.573257 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:55.573241045 +0000 UTC m=+68.947375319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : configmap "kube-rbac-proxy" not found Jul 16 08:55:53.858310 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:53.858226 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qhwlv_0c64883f-3ead-4052-bea8-c09675421988/dns-node-resolver/0.log" Jul 16 08:55:54.482491 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:54.482454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:55:54.482714 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:54.482635 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 08:55:54.482788 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:54.482761 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls podName:837b276c-808f-46c4-88c2-6c590b5d6c02 nodeName:}" failed. No retries permitted until 2026-07-16 08:56:26.482736261 +0000 UTC m=+99.856870539 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls") pod "dns-default-qkrk4" (UID: "837b276c-808f-46c4-88c2-6c590b5d6c02") : secret "dns-default-metrics-tls" not found Jul 16 08:55:54.684294 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:54.684254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:55:54.684294 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:54.684305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:55:54.684817 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:54.684416 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:54.684817 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:54.684483 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls podName:936e4c26-9180-4276-ab88-95b85db3653b nodeName:}" failed. No retries permitted until 2026-07-16 08:56:02.684463492 +0000 UTC m=+76.058597779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-54bw9" (UID: "936e4c26-9180-4276-ab88-95b85db3653b") : secret "cluster-monitoring-operator-tls" not found Jul 16 08:55:54.684817 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:54.684532 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 08:55:54.684817 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:54.684558 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd4d58779-5qp9b: secret "image-registry-tls" not found Jul 16 08:55:54.684817 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:54.684615 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls podName:662eb4c1-fee9-43f7-a560-09179fc18194 nodeName:}" failed. No retries permitted until 2026-07-16 08:56:02.68459902 +0000 UTC m=+76.058733289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls") pod "image-registry-bd4d58779-5qp9b" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194") : secret "image-registry-tls" not found Jul 16 08:55:55.059470 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:55.059438 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lpps8_928dda30-aac8-42e9-a376-9eb125067780/node-ca/0.log" Jul 16 08:55:55.455713 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:55.455680 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j6rmb_5abfceca-c1c1-42a0-bc57-ff208b0b63fd/serve-healthcheck-canary/0.log" Jul 16 08:55:55.594280 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:55.594240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:55.594479 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:55.594317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:55.594479 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:55.594402 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 08:55:55.594479 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:55.594453 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 08:55:55.594479 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:55.594478 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:59.594460741 +0000 UTC m=+72.968595021 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : secret "insights-runtime-extractor-tls" not found Jul 16 08:55:55.594670 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:55.594511 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:55:59.594497836 +0000 UTC m=+72.968632105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : configmap "kube-rbac-proxy" not found Jul 16 08:55:58.511951 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:58.511909 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pd2px" event={"ID":"491e030c-547f-4c64-be98-eb0c8313b800","Type":"ContainerStarted","Data":"3b2f3cf40f1eb6c91be63def6c833e66b6578ef395d1f4cebd08455a46902792"} Jul 16 08:55:58.513329 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:58.513304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kbb9c" event={"ID":"e21baeba-d058-495d-81b1-7a93f16372c3","Type":"ContainerStarted","Data":"ee3362845e2b538dbcb40e2fa1a86e5dca04c288b6ffe8c17a382524bd0e0a8f"} Jul 16 08:55:58.513459 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:58.513441 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:55:58.529529 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:58.529478 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pd2px" podStartSLOduration=66.191563422 podStartE2EDuration="1m11.529459187s" totalImagesPullingTime="5.337895765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:58.529059275 +0000 UTC m=+71.903193566" watchObservedRunningTime="2026-07-16 08:55:58.529459187 +0000 UTC m=+71.903593482" Jul 16 08:55:58.545335 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:58.545276 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kbb9c" podStartSLOduration=66.499822685 podStartE2EDuration="1m11.545258294s" totalImagesPullingTime="5.045435609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:55:58.544175811 +0000 UTC m=+71.918310096" watchObservedRunningTime="2026-07-16 08:55:58.545258294 +0000 UTC m=+71.919392584" Jul 16 08:55:59.629848 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:59.629800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:59.630306 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:55:59.629868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:55:59.630306 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:59.629922 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 08:55:59.630306 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:59.629968 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:56:07.629955843 +0000 UTC m=+81.004090111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : configmap "kube-rbac-proxy" not found Jul 16 08:55:59.630306 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:59.629983 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 08:55:59.630306 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:55:59.630045 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:56:07.630031199 +0000 UTC m=+81.004165473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : secret "insights-runtime-extractor-tls" not found Jul 16 08:56:02.755723 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:02.755682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls\") pod \"image-registry-bd4d58779-5qp9b\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:56:02.755723 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:02.755723 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:56:02.756295 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:02.755826 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 08:56:02.756295 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:02.755853 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bd4d58779-5qp9b: secret "image-registry-tls" not found Jul 16 08:56:02.756295 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:02.755830 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 16 08:56:02.756295 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:02.755911 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls podName:662eb4c1-fee9-43f7-a560-09179fc18194 nodeName:}" failed. No retries permitted until 2026-07-16 08:56:18.755896113 +0000 UTC m=+92.130030380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls") pod "image-registry-bd4d58779-5qp9b" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194") : secret "image-registry-tls" not found Jul 16 08:56:02.756295 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:02.755927 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls podName:936e4c26-9180-4276-ab88-95b85db3653b nodeName:}" failed. No retries permitted until 2026-07-16 08:56:18.755920947 +0000 UTC m=+92.130055215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-54bw9" (UID: "936e4c26-9180-4276-ab88-95b85db3653b") : secret "cluster-monitoring-operator-tls" not found Jul 16 08:56:07.699006 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:07.698843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:56:07.699006 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:07.698910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:56:07.699502 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:07.699010 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 08:56:07.699502 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:07.699094 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls podName:90534626-e36c-448a-93f8-159f941e95b8 nodeName:}" failed. No retries permitted until 2026-07-16 08:56:23.699074979 +0000 UTC m=+97.073209253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bmcmn" (UID: "90534626-e36c-448a-93f8-159f941e95b8") : secret "insights-runtime-extractor-tls" not found Jul 16 08:56:07.699589 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:07.699503 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90534626-e36c-448a-93f8-159f941e95b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:56:15.647320 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.647278 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bd4d58779-5qp9b"] Jul 16 08:56:15.647764 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:15.647519 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" podUID="662eb4c1-fee9-43f7-a560-09179fc18194" Jul 16 08:56:15.653297 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.653264 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n"] Jul 16 08:56:15.658049 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.658029 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.661146 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.661117 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 16 08:56:15.661146 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.661133 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 16 08:56:15.661146 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.661117 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 16 08:56:15.661389 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.661118 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 08:56:15.662014 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.661997 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 08:56:15.662093 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.661999 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 08:56:15.662954 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.662938 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 16 08:56:15.668276 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.668252 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n"] Jul 16 08:56:15.768875 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.768834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-hub\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.768875 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.768872 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmbr\" (UniqueName: \"kubernetes.io/projected/7571afdf-23da-4b6b-b74d-e1f3b0544c55-kube-api-access-tvmbr\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.769146 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.769002 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.769146 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.769050 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-ca\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.769146 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.769125 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7571afdf-23da-4b6b-b74d-e1f3b0544c55-additional-service-ca\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.769311 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.769183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.869653 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.869612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.869653 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.869663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-ca\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.869938 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.869699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7571afdf-23da-4b6b-b74d-e1f3b0544c55-additional-service-ca\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.869938 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.869716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.869938 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.869743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-hub\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.869938 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.869759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmbr\" (UniqueName: \"kubernetes.io/projected/7571afdf-23da-4b6b-b74d-e1f3b0544c55-kube-api-access-tvmbr\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.870617 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.870589 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7571afdf-23da-4b6b-b74d-e1f3b0544c55-additional-service-ca\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.872498 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.872468 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-ca\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.872618 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.872565 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.872680 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.872662 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.872718 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.872679 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7571afdf-23da-4b6b-b74d-e1f3b0544c55-hub\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.883550 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.883516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmbr\" (UniqueName: \"kubernetes.io/projected/7571afdf-23da-4b6b-b74d-e1f3b0544c55-kube-api-access-tvmbr\") pod \"cluster-proxy-proxy-agent-647c649686-kp96n\" (UID: \"7571afdf-23da-4b6b-b74d-e1f3b0544c55\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:15.984094 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:15.983997 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" Jul 16 08:56:16.122574 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.122535 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n"] Jul 16 08:56:16.125950 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:56:16.125920 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7571afdf_23da_4b6b_b74d_e1f3b0544c55.slice/crio-8520c61755542b385e2d23b784063cdea49a1903a238705a4320b2b2085c2344 WatchSource:0}: Error finding container 8520c61755542b385e2d23b784063cdea49a1903a238705a4320b2b2085c2344: Status 404 returned error can't find the container with id 8520c61755542b385e2d23b784063cdea49a1903a238705a4320b2b2085c2344 Jul 16 08:56:16.568080 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.568031 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" event={"ID":"7571afdf-23da-4b6b-b74d-e1f3b0544c55","Type":"ContainerStarted","Data":"8520c61755542b385e2d23b784063cdea49a1903a238705a4320b2b2085c2344"} Jul 16 08:56:16.568080 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.568058 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:56:16.576165 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.576121 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:56:16.677527 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.677493 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-image-registry-private-configuration\") pod \"662eb4c1-fee9-43f7-a560-09179fc18194\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " Jul 16 08:56:16.678028 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.677538 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-installation-pull-secrets\") pod \"662eb4c1-fee9-43f7-a560-09179fc18194\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " Jul 16 08:56:16.678028 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.677585 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-bound-sa-token\") pod \"662eb4c1-fee9-43f7-a560-09179fc18194\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " Jul 16 08:56:16.678028 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.677625 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-trusted-ca\") pod \"662eb4c1-fee9-43f7-a560-09179fc18194\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " Jul 16 08:56:16.678028 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.677656 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-registry-certificates\") pod \"662eb4c1-fee9-43f7-a560-09179fc18194\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " Jul 16 08:56:16.678028 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.677744 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662eb4c1-fee9-43f7-a560-09179fc18194-ca-trust-extracted\") pod \"662eb4c1-fee9-43f7-a560-09179fc18194\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " Jul 16 08:56:16.678028 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.677813 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4vw9\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-kube-api-access-k4vw9\") pod \"662eb4c1-fee9-43f7-a560-09179fc18194\" (UID: \"662eb4c1-fee9-43f7-a560-09179fc18194\") " Jul 16 08:56:16.678028 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.678001 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/662eb4c1-fee9-43f7-a560-09179fc18194-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "662eb4c1-fee9-43f7-a560-09179fc18194" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 08:56:16.678395 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.678052 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "662eb4c1-fee9-43f7-a560-09179fc18194" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 08:56:16.678395 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.678176 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "662eb4c1-fee9-43f7-a560-09179fc18194" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 08:56:16.678395 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.678192 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-registry-certificates\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:56:16.678395 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.678212 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/662eb4c1-fee9-43f7-a560-09179fc18194-ca-trust-extracted\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:56:16.680224 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.680113 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "662eb4c1-fee9-43f7-a560-09179fc18194" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:56:16.680364 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.680237 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-kube-api-access-k4vw9" (OuterVolumeSpecName: "kube-api-access-k4vw9") pod "662eb4c1-fee9-43f7-a560-09179fc18194" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194"). InnerVolumeSpecName "kube-api-access-k4vw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 08:56:16.680364 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.680273 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "662eb4c1-fee9-43f7-a560-09179fc18194" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:56:16.680364 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.680321 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "662eb4c1-fee9-43f7-a560-09179fc18194" (UID: "662eb4c1-fee9-43f7-a560-09179fc18194"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 08:56:16.779914 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.779875 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-image-registry-private-configuration\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:56:16.779914 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.779909 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/662eb4c1-fee9-43f7-a560-09179fc18194-installation-pull-secrets\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:56:16.779914 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.779921 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-bound-sa-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:56:16.780271 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.779938 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/662eb4c1-fee9-43f7-a560-09179fc18194-trusted-ca\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:56:16.780271 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:16.779949 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4vw9\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-kube-api-access-k4vw9\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:56:17.571467 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:17.571431 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bd4d58779-5qp9b" Jul 16 08:56:17.604782 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:17.604740 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bd4d58779-5qp9b"] Jul 16 08:56:17.611544 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:17.611506 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bd4d58779-5qp9b"] Jul 16 08:56:17.692972 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:17.692935 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/662eb4c1-fee9-43f7-a560-09179fc18194-registry-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:56:18.803149 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:18.803095 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:56:18.806322 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:18.806289 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/936e4c26-9180-4276-ab88-95b85db3653b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-54bw9\" (UID: \"936e4c26-9180-4276-ab88-95b85db3653b\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:56:18.898101 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:18.898064 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qlq58\"" Jul 16 08:56:18.905738 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:18.905700 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" Jul 16 08:56:19.054642 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:19.054541 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="662eb4c1-fee9-43f7-a560-09179fc18194" path="/var/lib/kubelet/pods/662eb4c1-fee9-43f7-a560-09179fc18194/volumes" Jul 16 08:56:19.604626 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:19.604441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9"] Jul 16 08:56:19.607826 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:56:19.607794 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936e4c26_9180_4276_ab88_95b85db3653b.slice/crio-a2fac0efd89150f7cafaf852b300b406fd2f549a079f157902108159206a49f5 WatchSource:0}: Error finding container a2fac0efd89150f7cafaf852b300b406fd2f549a079f157902108159206a49f5: Status 404 returned error can't find the container with id a2fac0efd89150f7cafaf852b300b406fd2f549a079f157902108159206a49f5 Jul 16 08:56:20.586418 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:20.586378 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" event={"ID":"936e4c26-9180-4276-ab88-95b85db3653b","Type":"ContainerStarted","Data":"a2fac0efd89150f7cafaf852b300b406fd2f549a079f157902108159206a49f5"} Jul 16 08:56:20.588464 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:20.588432 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" event={"ID":"7571afdf-23da-4b6b-b74d-e1f3b0544c55","Type":"ContainerStarted","Data":"257d3035b5e07943def10b4998253add63c1bc335128f14d483f53a2da54fe97"} Jul 16 08:56:20.588618 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:20.588482 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" event={"ID":"7571afdf-23da-4b6b-b74d-e1f3b0544c55","Type":"ContainerStarted","Data":"ac64975a89ab2301f519a3e2759a35354d8887f5ac34a204ffd947fa241e7ff4"} Jul 16 08:56:20.588618 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:20.588501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" event={"ID":"7571afdf-23da-4b6b-b74d-e1f3b0544c55","Type":"ContainerStarted","Data":"778f0f2f8c26ca6336611587e10d4e86010289f88b8522cd82703eb32743ecf2"} Jul 16 08:56:20.608678 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:20.608618 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-647c649686-kp96n" podStartSLOduration=2.195477375 podStartE2EDuration="5.608598713s" totalImagesPullingTime="3.413121338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:56:20.607858479 +0000 UTC m=+93.981992769" watchObservedRunningTime="2026-07-16 08:56:20.608598713 +0000 UTC m=+93.982733003" Jul 16 08:56:23.601789 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:23.601751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" event={"ID":"936e4c26-9180-4276-ab88-95b85db3653b","Type":"ContainerStarted","Data":"cbbc6bf65f7cb520b47b1964d80c5c5292653ad24d35f1cc157f951742ce1c27"} Jul 16 08:56:23.619013 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:23.618951 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-54bw9" podStartSLOduration=34.572260141 podStartE2EDuration="37.618930278s" totalImagesPullingTime="3.046670137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:56:23.617995017 +0000 UTC m=+96.992129307" watchObservedRunningTime="2026-07-16 08:56:23.618930278 +0000 UTC m=+96.993064568" Jul 16 08:56:23.744729 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:23.744687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:56:23.747370 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:23.747327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90534626-e36c-448a-93f8-159f941e95b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bmcmn\" (UID: \"90534626-e36c-448a-93f8-159f941e95b8\") " pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:56:23.989675 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:23.989639 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bmcmn" Jul 16 08:56:24.128975 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:24.128931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bmcmn"] Jul 16 08:56:24.132047 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:56:24.132003 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90534626_e36c_448a_93f8_159f941e95b8.slice/crio-f8fcadfb26a786493c5aa752f234d10ea130bbab4f1965399cc7d77a00545b77 WatchSource:0}: Error finding container f8fcadfb26a786493c5aa752f234d10ea130bbab4f1965399cc7d77a00545b77: Status 404 returned error can't find the container with id f8fcadfb26a786493c5aa752f234d10ea130bbab4f1965399cc7d77a00545b77 Jul 16 08:56:24.606978 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:24.606940 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmcmn" event={"ID":"90534626-e36c-448a-93f8-159f941e95b8","Type":"ContainerStarted","Data":"f8fcadfb26a786493c5aa752f234d10ea130bbab4f1965399cc7d77a00545b77"} Jul 16 08:56:26.569534 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:26.569427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:56:26.572333 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:26.572304 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/837b276c-808f-46c4-88c2-6c590b5d6c02-metrics-tls\") pod \"dns-default-qkrk4\" (UID: \"837b276c-808f-46c4-88c2-6c590b5d6c02\") " pod="openshift-dns/dns-default-qkrk4" Jul 16 08:56:26.828175 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:26.828058 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wm6pc\"" Jul 16 08:56:26.836433 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:26.836390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qkrk4" Jul 16 08:56:29.519514 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:29.519479 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kbb9c" Jul 16 08:56:30.171297 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:30.171261 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qkrk4"] Jul 16 08:56:30.174434 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:56:30.174403 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837b276c_808f_46c4_88c2_6c590b5d6c02.slice/crio-6090fb266646fe17a8583c23c50fe2c101f13c7512d37e2427ebf1648c397f35 WatchSource:0}: Error finding container 6090fb266646fe17a8583c23c50fe2c101f13c7512d37e2427ebf1648c397f35: Status 404 returned error can't find the container with id 6090fb266646fe17a8583c23c50fe2c101f13c7512d37e2427ebf1648c397f35 Jul 16 08:56:30.628784 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:30.628726 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmcmn" event={"ID":"90534626-e36c-448a-93f8-159f941e95b8","Type":"ContainerStarted","Data":"c9d1e556d5a5b2f559d2da91f6fcc418244bc42313786ed41d444cf4c7dd52ee"} Jul 16 08:56:30.629849 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:30.629822 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qkrk4" event={"ID":"837b276c-808f-46c4-88c2-6c590b5d6c02","Type":"ContainerStarted","Data":"6090fb266646fe17a8583c23c50fe2c101f13c7512d37e2427ebf1648c397f35"} Jul 16 08:56:32.584416 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.584378 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq"] Jul 16 08:56:32.587825 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.587796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.590797 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.590772 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 08:56:32.590985 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.590776 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 08:56:32.590985 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.590889 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gpnbx\"" Jul 16 08:56:32.590985 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.590910 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 08:56:32.595689 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.595371 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-px2qw"] Jul 16 08:56:32.600407 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.600385 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.603307 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.603283 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 08:56:32.603472 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.603296 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 08:56:32.603472 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.603351 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 08:56:32.603472 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.603462 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6fx9n\"" Jul 16 08:56:32.607986 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.607894 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq"] Jul 16 08:56:32.622204 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.622145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e806b6a-741c-4a34-be65-c965ce89ec25-root\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.622726 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.622526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-textfile\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.622869 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.622767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-tls\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.622869 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.622831 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.622988 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.622869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-wtmp\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.622988 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.622897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph66k\" (UniqueName: \"kubernetes.io/projected/1e806b6a-741c-4a34-be65-c965ce89ec25-kube-api-access-ph66k\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.622988 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.622919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.623129 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.623022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-accelerators-collector-config\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.623129 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.623062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.623129 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.623091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.623292 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.623221 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhcv\" (UniqueName: \"kubernetes.io/projected/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-kube-api-access-qvhcv\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.623368 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.623351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e806b6a-741c-4a34-be65-c965ce89ec25-metrics-client-ca\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.623428 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.623404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e806b6a-741c-4a34-be65-c965ce89ec25-sys\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.724720 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.724928 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhcv\" (UniqueName: \"kubernetes.io/projected/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-kube-api-access-qvhcv\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.724928 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e806b6a-741c-4a34-be65-c965ce89ec25-metrics-client-ca\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.724928 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e806b6a-741c-4a34-be65-c965ce89ec25-sys\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.724928 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e806b6a-741c-4a34-be65-c965ce89ec25-root\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.724928 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-textfile\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725233 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-tls\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725233 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.725233 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-wtmp\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725233 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.724996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e806b6a-741c-4a34-be65-c965ce89ec25-sys\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725233 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.725024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph66k\" (UniqueName: \"kubernetes.io/projected/1e806b6a-741c-4a34-be65-c965ce89ec25-kube-api-access-ph66k\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725233 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.725052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.725233 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.725071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1e806b6a-741c-4a34-be65-c965ce89ec25-root\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725233 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.725088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-accelerators-collector-config\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725233 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.725115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.725645 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.725262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-wtmp\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725645 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:32.725380 2574 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 16 08:56:32.725645 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:32.725475 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-openshift-state-metrics-tls podName:240fc7fe-4395-4dbc-b0ea-f9b6da29938e nodeName:}" failed. No retries permitted until 2026-07-16 08:56:33.225450465 +0000 UTC m=+106.599584736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-n2vfq" (UID: "240fc7fe-4395-4dbc-b0ea-f9b6da29938e") : secret "openshift-state-metrics-tls" not found Jul 16 08:56:32.725645 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.725498 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-textfile\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725645 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:32.725619 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 08:56:32.725881 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:56:32.725699 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-tls podName:1e806b6a-741c-4a34-be65-c965ce89ec25 nodeName:}" failed. No retries permitted until 2026-07-16 08:56:33.225680562 +0000 UTC m=+106.599814830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-tls") pod "node-exporter-px2qw" (UID: "1e806b6a-741c-4a34-be65-c965ce89ec25") : secret "node-exporter-tls" not found Jul 16 08:56:32.725881 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.725747 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e806b6a-741c-4a34-be65-c965ce89ec25-metrics-client-ca\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.725997 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.725944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-accelerators-collector-config\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.726331 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.726301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.728043 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.728012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.728196 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.728141 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:32.736867 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.736827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph66k\" (UniqueName: \"kubernetes.io/projected/1e806b6a-741c-4a34-be65-c965ce89ec25-kube-api-access-ph66k\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:32.738757 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:32.738701 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhcv\" (UniqueName: \"kubernetes.io/projected/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-kube-api-access-qvhcv\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:33.229284 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.229242 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:33.229586 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.229390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-tls\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:33.232211 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.232177 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1e806b6a-741c-4a34-be65-c965ce89ec25-node-exporter-tls\") pod \"node-exporter-px2qw\" (UID: \"1e806b6a-741c-4a34-be65-c965ce89ec25\") " pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:33.232211 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.232207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/240fc7fe-4395-4dbc-b0ea-f9b6da29938e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-n2vfq\" (UID: \"240fc7fe-4395-4dbc-b0ea-f9b6da29938e\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:33.499311 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.499197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" Jul 16 08:56:33.512386 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.512336 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-px2qw" Jul 16 08:56:33.735827 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.735776 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-tdvbs"] Jul 16 08:56:33.747304 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.747269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-tdvbs" Jul 16 08:56:33.748061 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.748027 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-tdvbs"] Jul 16 08:56:33.750582 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.750216 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4ljf9\"" Jul 16 08:56:33.750582 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.750338 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 08:56:33.750582 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.750222 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 08:56:33.834992 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.834940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2nnz\" (UniqueName: \"kubernetes.io/projected/2d0c6c73-8026-446d-95bd-e8515a8552e4-kube-api-access-t2nnz\") pod \"downloads-68c9cbb497-tdvbs\" (UID: \"2d0c6c73-8026-446d-95bd-e8515a8552e4\") " pod="openshift-console/downloads-68c9cbb497-tdvbs" Jul 16 08:56:33.937264 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.937226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2nnz\" (UniqueName: \"kubernetes.io/projected/2d0c6c73-8026-446d-95bd-e8515a8552e4-kube-api-access-t2nnz\") pod \"downloads-68c9cbb497-tdvbs\" (UID: \"2d0c6c73-8026-446d-95bd-e8515a8552e4\") " pod="openshift-console/downloads-68c9cbb497-tdvbs" Jul 16 08:56:33.949243 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:33.949210 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2nnz\" (UniqueName: \"kubernetes.io/projected/2d0c6c73-8026-446d-95bd-e8515a8552e4-kube-api-access-t2nnz\") pod \"downloads-68c9cbb497-tdvbs\" (UID: \"2d0c6c73-8026-446d-95bd-e8515a8552e4\") " pod="openshift-console/downloads-68c9cbb497-tdvbs" Jul 16 08:56:34.061818 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:34.061713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-tdvbs" Jul 16 08:56:34.656208 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:56:34.656084 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e806b6a_741c_4a34_be65_c965ce89ec25.slice/crio-6427a067a9288c9323bf7382eec3328af5290a71d4507c7783a88e8b144ea510 WatchSource:0}: Error finding container 6427a067a9288c9323bf7382eec3328af5290a71d4507c7783a88e8b144ea510: Status 404 returned error can't find the container with id 6427a067a9288c9323bf7382eec3328af5290a71d4507c7783a88e8b144ea510 Jul 16 08:56:34.819982 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:34.819831 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-tdvbs"] Jul 16 08:56:34.825106 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:56:34.825021 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0c6c73_8026_446d_95bd_e8515a8552e4.slice/crio-9607562ee87c91eac3972393626d19bb51d78680ae86e828b3f9e98d670882ed WatchSource:0}: Error finding container 9607562ee87c91eac3972393626d19bb51d78680ae86e828b3f9e98d670882ed: Status 404 returned error can't find the container with id 9607562ee87c91eac3972393626d19bb51d78680ae86e828b3f9e98d670882ed Jul 16 08:56:34.844041 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:34.844003 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq"] Jul 16 08:56:34.847725 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:56:34.847693 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240fc7fe_4395_4dbc_b0ea_f9b6da29938e.slice/crio-dd92384c7255c80969590836782c910f0fcb05b8427b052d72282b50ea5c28c4 WatchSource:0}: Error finding container dd92384c7255c80969590836782c910f0fcb05b8427b052d72282b50ea5c28c4: Status 404 returned error can't find the container with id dd92384c7255c80969590836782c910f0fcb05b8427b052d72282b50ea5c28c4 Jul 16 08:56:35.657870 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.656349 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmcmn" event={"ID":"90534626-e36c-448a-93f8-159f941e95b8","Type":"ContainerStarted","Data":"6549ea53a70b252cbee32e17450b1439f7a6898f6ee3fcbf30e3d7f313d6b56d"} Jul 16 08:56:35.659264 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.659172 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px2qw" event={"ID":"1e806b6a-741c-4a34-be65-c965ce89ec25","Type":"ContainerStarted","Data":"6427a067a9288c9323bf7382eec3328af5290a71d4507c7783a88e8b144ea510"} Jul 16 08:56:35.662462 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.662430 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qkrk4" event={"ID":"837b276c-808f-46c4-88c2-6c590b5d6c02","Type":"ContainerStarted","Data":"240d69992b17956d71413e0f418e915ad337278c1d6b992e40a0da36bdc6c9ab"} Jul 16 08:56:35.662621 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.662497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qkrk4" event={"ID":"837b276c-808f-46c4-88c2-6c590b5d6c02","Type":"ContainerStarted","Data":"a23162a02c2f38c756e9624666f2117d265cdec588e0d3e5c86ea78bd3c77ce7"} Jul 16 08:56:35.663444 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.663424 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qkrk4" Jul 16 08:56:35.665910 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.665879 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-tdvbs" event={"ID":"2d0c6c73-8026-446d-95bd-e8515a8552e4","Type":"ContainerStarted","Data":"9607562ee87c91eac3972393626d19bb51d78680ae86e828b3f9e98d670882ed"} Jul 16 08:56:35.669607 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.669475 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" event={"ID":"240fc7fe-4395-4dbc-b0ea-f9b6da29938e","Type":"ContainerStarted","Data":"014a7977c5c1ccf9da5c95db08cfe3cc5aa4b5e3287a35d8315b8a9c587b7e18"} Jul 16 08:56:35.669607 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.669541 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" event={"ID":"240fc7fe-4395-4dbc-b0ea-f9b6da29938e","Type":"ContainerStarted","Data":"69be0c5b300df5b6cf364b13818c2e1061a871ad43ad7bf4183685083c527895"} Jul 16 08:56:35.669607 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.669571 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" event={"ID":"240fc7fe-4395-4dbc-b0ea-f9b6da29938e","Type":"ContainerStarted","Data":"dd92384c7255c80969590836782c910f0fcb05b8427b052d72282b50ea5c28c4"} Jul 16 08:56:35.686064 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:35.685819 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qkrk4" podStartSLOduration=69.213673927 podStartE2EDuration="1m13.685791654s" totalImagesPullingTime="4.472117727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:56:35.684182068 +0000 UTC m=+109.058316362" watchObservedRunningTime="2026-07-16 08:56:35.685791654 +0000 UTC m=+109.059925945" Jul 16 08:56:36.680863 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:36.680547 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" event={"ID":"240fc7fe-4395-4dbc-b0ea-f9b6da29938e","Type":"ContainerStarted","Data":"7c81b26f5d140b537773309ebeea92f98034a4cda6039d1e6cd147885fe5941c"} Jul 16 08:56:36.682656 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:36.682614 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e806b6a-741c-4a34-be65-c965ce89ec25" containerID="21ef09aca4423d0193759edc33404a4bffe7195babcfa5e37f5e1abb64c0c751" exitCode=0 Jul 16 08:56:36.682870 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:36.682844 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px2qw" event={"ID":"1e806b6a-741c-4a34-be65-c965ce89ec25","Type":"ContainerDied","Data":"21ef09aca4423d0193759edc33404a4bffe7195babcfa5e37f5e1abb64c0c751"} Jul 16 08:56:36.703220 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:36.703131 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-n2vfq" podStartSLOduration=3.138333551 podStartE2EDuration="4.70311244s" totalImagesPullingTime="1.564778889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:56:36.700424791 +0000 UTC m=+110.074559081" watchObservedRunningTime="2026-07-16 08:56:36.70311244 +0000 UTC m=+110.077246730" Jul 16 08:56:37.691405 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:37.691241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px2qw" event={"ID":"1e806b6a-741c-4a34-be65-c965ce89ec25","Type":"ContainerStarted","Data":"7cfa18c393f9b70f9b775b34b8012f45a6debf9d92bb81f767337d8e3c95c72e"} Jul 16 08:56:37.691405 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:37.691315 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px2qw" event={"ID":"1e806b6a-741c-4a34-be65-c965ce89ec25","Type":"ContainerStarted","Data":"96fa6e27340da3d494e182a1c944ed0ad28f1f01d9ef9981dd7874283cb57a27"} Jul 16 08:56:37.712857 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:37.712792 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-px2qw" podStartSLOduration=4.465164236 podStartE2EDuration="5.712768593s" totalImagesPullingTime="1.156915621s" totalInitContainerRuntime="90.688736ms" isStatefulPod=true podCreationTimestamp="2026-07-16 08:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:56:37.711485494 +0000 UTC m=+111.085619786" watchObservedRunningTime="2026-07-16 08:56:37.712768593 +0000 UTC m=+111.086902883" Jul 16 08:56:38.881444 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.881399 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 08:56:38.911371 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.911330 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 08:56:38.911594 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.911492 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.914500 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.914463 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 08:56:38.915632 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915095 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 08:56:38.915632 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915215 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 08:56:38.915632 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915262 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 08:56:38.915632 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915277 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 08:56:38.915632 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915095 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 08:56:38.915632 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915395 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2rrv8aqba1824\"" Jul 16 08:56:38.915632 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915524 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 08:56:38.915632 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915567 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 08:56:38.916393 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915835 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 08:56:38.916393 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.915988 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vzz9p\"" Jul 16 08:56:38.916393 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.916000 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 08:56:38.916581 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.916461 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 08:56:38.916668 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.916641 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 08:56:38.917016 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.916902 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 08:56:38.918941 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.918876 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 08:56:38.922481 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.922433 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 08:56:38.989577 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.989544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.989785 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.989684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.989785 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.989718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.989926 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.989815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.989926 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.989860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.989926 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.989891 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.989926 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.989918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990110 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.989973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990110 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990110 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990060 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990110 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990324 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n76wn\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-kube-api-access-n76wn\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990324 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-config\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990324 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990324 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990324 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990324 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990281 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:38.990609 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:38.990387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.091858 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.091858 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.091858 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091692 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.091858 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.091858 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.091858 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.091858 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76wn\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-kube-api-access-n76wn\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-config\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.091974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.092004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.092027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.092053 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.092079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.092106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092653 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.092509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.092848 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.092826 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.093080 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.093055 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.093175 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.093070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.093428 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.093396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.093536 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.093471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.093536 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.093508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.093646 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.093569 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.094716 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.094676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.096676 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.096052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.096676 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.096505 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.096929 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.096903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.097907 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.097858 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.099725 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.099679 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.099840 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.099786 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.099840 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.099812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.100360 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.100299 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-config-out\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.100491 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.100405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.100491 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.100469 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.100669 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.100643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.100789 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.100739 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-web-config\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.101311 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.101292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-config\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.101699 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.101678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.105300 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.105256 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n76wn\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-kube-api-access-n76wn\") pod \"prometheus-k8s-0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.225701 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.225587 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:56:39.402273 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.402208 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 08:56:39.405647 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:56:39.405609 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfde344_5055_4239_a97f_3a1f991d8ac0.slice/crio-c46e92d34ffbeb378af0643801a6bea87e4949f01656ec618a3d13713184d880 WatchSource:0}: Error finding container c46e92d34ffbeb378af0643801a6bea87e4949f01656ec618a3d13713184d880: Status 404 returned error can't find the container with id c46e92d34ffbeb378af0643801a6bea87e4949f01656ec618a3d13713184d880 Jul 16 08:56:39.704127 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.704076 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bmcmn" event={"ID":"90534626-e36c-448a-93f8-159f941e95b8","Type":"ContainerStarted","Data":"3eed33fbcf100cd05906f173e409955e284a381bce05358db0c4d36a2b37b92d"} Jul 16 08:56:39.705551 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.705516 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerStarted","Data":"c46e92d34ffbeb378af0643801a6bea87e4949f01656ec618a3d13713184d880"} Jul 16 08:56:39.724351 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:39.724288 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bmcmn" podStartSLOduration=34.046726592 podStartE2EDuration="48.72426968s" totalImagesPullingTime="14.677543088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 08:55:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:56:39.723342911 +0000 UTC m=+113.097477208" watchObservedRunningTime="2026-07-16 08:56:39.72426968 +0000 UTC m=+113.098403979" Jul 16 08:56:41.717476 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:41.717428 2574 generic.go:356] "Generic (PLEG): container finished" podID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerID="cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299" exitCode=0 Jul 16 08:56:41.718002 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:41.717477 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerDied","Data":"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299"} Jul 16 08:56:46.694722 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:46.694690 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qkrk4" Jul 16 08:56:52.770070 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:52.770028 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerStarted","Data":"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972"} Jul 16 08:56:53.779445 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:53.779382 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerStarted","Data":"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381"} Jul 16 08:56:53.782101 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:53.782007 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-tdvbs" event={"ID":"2d0c6c73-8026-446d-95bd-e8515a8552e4","Type":"ContainerStarted","Data":"064103da6f8789cc655caacad58c3cb278f4822cdf74c32ef31bf6e8a1f5bff3"} Jul 16 08:56:53.782619 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:53.782484 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-tdvbs" Jul 16 08:56:53.797735 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:53.797690 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-tdvbs" Jul 16 08:56:53.802318 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:53.802248 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-tdvbs" podStartSLOduration=2.851911325 podStartE2EDuration="20.802219182s" totalImagesPullingTime="17.950307857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:56:53.799994853 +0000 UTC m=+127.174129143" watchObservedRunningTime="2026-07-16 08:56:53.802219182 +0000 UTC m=+127.176353519" Jul 16 08:56:55.812861 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:55.812670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerStarted","Data":"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7"} Jul 16 08:56:55.812861 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:55.812793 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerStarted","Data":"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60"} Jul 16 08:56:55.874647 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:55.874597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:56:55.877755 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:55.877711 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7689dfa7-1f38-485d-a5e5-1fbea1c74f57-metrics-certs\") pod \"network-metrics-daemon-7zblf\" (UID: \"7689dfa7-1f38-485d-a5e5-1fbea1c74f57\") " pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:56:56.168441 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:56.168400 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s7ks4\"" Jul 16 08:56:56.176667 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:56.176617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7zblf" Jul 16 08:56:56.371822 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:56.371780 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7zblf"] Jul 16 08:56:56.376655 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:56:56.376557 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7689dfa7_1f38_485d_a5e5_1fbea1c74f57.slice/crio-8aedbc1f4595749fb44c60dcf72301ad482eb4fa8b0e3bd8a675bfb763531398 WatchSource:0}: Error finding container 8aedbc1f4595749fb44c60dcf72301ad482eb4fa8b0e3bd8a675bfb763531398: Status 404 returned error can't find the container with id 8aedbc1f4595749fb44c60dcf72301ad482eb4fa8b0e3bd8a675bfb763531398 Jul 16 08:56:56.824572 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:56.824533 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerStarted","Data":"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689"} Jul 16 08:56:56.825089 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:56.824620 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerStarted","Data":"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545"} Jul 16 08:56:56.827407 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:56.827305 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7zblf" event={"ID":"7689dfa7-1f38-485d-a5e5-1fbea1c74f57","Type":"ContainerStarted","Data":"8aedbc1f4595749fb44c60dcf72301ad482eb4fa8b0e3bd8a675bfb763531398"} Jul 16 08:56:56.857195 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:56.856870 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.061626007 podStartE2EDuration="18.856841703s" totalImagesPullingTime="14.773684837s" totalInitContainerRuntime="21.530859ms" isStatefulPod=false podCreationTimestamp="2026-07-16 08:56:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:56:56.854505687 +0000 UTC m=+130.228639989" watchObservedRunningTime="2026-07-16 08:56:56.856841703 +0000 UTC m=+130.230975994" Jul 16 08:56:58.839859 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:58.839814 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7zblf" event={"ID":"7689dfa7-1f38-485d-a5e5-1fbea1c74f57","Type":"ContainerStarted","Data":"48a74f5419cfaf144859427610bd14aba84a5a10c895864d289f3c1a0cc8c0b0"} Jul 16 08:56:58.840417 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:58.839941 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7zblf" event={"ID":"7689dfa7-1f38-485d-a5e5-1fbea1c74f57","Type":"ContainerStarted","Data":"6fc986bf4525996f5fb42db5b9b14c67609ce5c018674c4b2cb4fe233b011127"} Jul 16 08:56:58.858042 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:58.857966 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7zblf" podStartSLOduration=130.356270398 podStartE2EDuration="2m11.857939512s" totalImagesPullingTime="1.501669114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 08:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:56:58.857099862 +0000 UTC m=+132.231234152" watchObservedRunningTime="2026-07-16 08:56:58.857939512 +0000 UTC m=+132.232073809" Jul 16 08:56:59.226430 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:56:59.226379 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:06.876798 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:06.876759 2574 generic.go:356] "Generic (PLEG): container finished" podID="f001e84b-5fab-4319-814e-19e40d8cc591" containerID="5ffea6d052e1aa22ef23f9170b3cbec0ac7cfe748ae318577063c41768294ba1" exitCode=0 Jul 16 08:57:06.877399 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:06.876834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" event={"ID":"f001e84b-5fab-4319-814e-19e40d8cc591","Type":"ContainerDied","Data":"5ffea6d052e1aa22ef23f9170b3cbec0ac7cfe748ae318577063c41768294ba1"} Jul 16 08:57:06.877399 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:06.877321 2574 scope.go:117] "RemoveContainer" containerID="5ffea6d052e1aa22ef23f9170b3cbec0ac7cfe748ae318577063c41768294ba1" Jul 16 08:57:07.882702 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:07.882661 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-x9ch5" event={"ID":"f001e84b-5fab-4319-814e-19e40d8cc591","Type":"ContainerStarted","Data":"49f8e98eb3c93bed07eb598447454c66fe33d38af8511043c7944e90b4ca9582"} Jul 16 08:57:10.897628 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:10.897590 2574 generic.go:356] "Generic (PLEG): container finished" podID="4526fa9c-78bc-463d-8244-cf63b0c783bb" containerID="9987df9d6ac41f1f6d753d8f8b81b47e44700f2db58c96ebd56109b0006f3f36" exitCode=0 Jul 16 08:57:10.898122 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:10.897669 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" event={"ID":"4526fa9c-78bc-463d-8244-cf63b0c783bb","Type":"ContainerDied","Data":"9987df9d6ac41f1f6d753d8f8b81b47e44700f2db58c96ebd56109b0006f3f36"} Jul 16 08:57:10.898239 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:10.898136 2574 scope.go:117] "RemoveContainer" containerID="9987df9d6ac41f1f6d753d8f8b81b47e44700f2db58c96ebd56109b0006f3f36" Jul 16 08:57:11.903524 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:11.903489 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-j4c9z" event={"ID":"4526fa9c-78bc-463d-8244-cf63b0c783bb","Type":"ContainerStarted","Data":"3424be16d0ec9f9d602c0fc8d4347a150e457d52b9b8d13c36c8253279a113dd"} Jul 16 08:57:16.925035 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:16.924984 2574 generic.go:356] "Generic (PLEG): container finished" podID="019c8aea-66ce-4497-bc6a-550bdefad29c" containerID="d9b17b02454fa564fa62426b58dd59144d1b33c6ef139aa32d2b9a995133cfa8" exitCode=0 Jul 16 08:57:16.925534 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:16.925060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" event={"ID":"019c8aea-66ce-4497-bc6a-550bdefad29c","Type":"ContainerDied","Data":"d9b17b02454fa564fa62426b58dd59144d1b33c6ef139aa32d2b9a995133cfa8"} Jul 16 08:57:16.925585 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:16.925567 2574 scope.go:117] "RemoveContainer" containerID="d9b17b02454fa564fa62426b58dd59144d1b33c6ef139aa32d2b9a995133cfa8" Jul 16 08:57:17.931879 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:17.931838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-8g4xv" event={"ID":"019c8aea-66ce-4497-bc6a-550bdefad29c","Type":"ContainerStarted","Data":"e5983acdae34ffed0521c3e6d44d5d4260ecb5cbdf45f823d962aebe7aaa4b17"} Jul 16 08:57:39.226724 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:39.226605 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:39.284831 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:39.284798 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:40.046109 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:40.046077 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:57.267209 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.267169 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 08:57:57.267790 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.267485 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="prometheus" containerID="cri-o://6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" gracePeriod=600 Jul 16 08:57:57.267790 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.267506 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy" containerID="cri-o://c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" gracePeriod=600 Jul 16 08:57:57.267790 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.267526 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="thanos-sidecar" containerID="cri-o://a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" gracePeriod=600 Jul 16 08:57:57.267790 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.267566 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="config-reloader" containerID="cri-o://8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" gracePeriod=600 Jul 16 08:57:57.267790 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.267526 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" gracePeriod=600 Jul 16 08:57:57.267790 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.267617 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy-web" containerID="cri-o://49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" gracePeriod=600 Jul 16 08:57:57.522367 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.522288 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:57.668899 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.668846 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-grpc-tls\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.668899 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.668900 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-2\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669244 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.668936 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n76wn\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-kube-api-access-n76wn\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669244 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.668966 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-thanos-prometheus-http-client-file\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669244 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669007 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-1\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669244 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669030 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-metrics-client-certs\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669244 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669217 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-web-config\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669501 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669269 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-kube-rbac-proxy\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669501 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669301 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-config\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669501 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669347 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669501 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669378 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-tls-assets\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669501 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669410 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669501 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669442 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-tls\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669501 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669449 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 08:57:57.669501 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669484 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-metrics-client-ca\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669900 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669514 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-config-out\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669900 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669566 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-trusted-ca-bundle\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669900 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669591 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-kubelet-serving-ca-bundle\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669900 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669615 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-0\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669900 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669654 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-serving-certs-ca-bundle\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.669900 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669708 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-db\") pod \"fcfde344-5055-4239-a97f-3a1f991d8ac0\" (UID: \"fcfde344-5055-4239-a97f-3a1f991d8ac0\") " Jul 16 08:57:57.670236 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.669677 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 08:57:57.671405 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.671356 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 08:57:57.672211 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.672133 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:57:57.672554 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.672520 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:57:57.672670 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.672567 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.672670 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.672589 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-db\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.672670 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.672605 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.672670 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.672619 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.673338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.673225 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:57:57.673338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.673257 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:57:57.673338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.673300 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-kube-api-access-n76wn" (OuterVolumeSpecName: "kube-api-access-n76wn") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "kube-api-access-n76wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 08:57:57.674888 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.673737 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 08:57:57.674888 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.674112 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 08:57:57.674888 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.674540 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 08:57:57.674888 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.674538 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 08:57:57.674888 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.674831 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:57:57.675248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.675215 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 08:57:57.676070 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.675800 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-config" (OuterVolumeSpecName: "config") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:57:57.676070 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.676031 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 08:57:57.676263 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.676101 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:57:57.677069 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.676385 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-config-out" (OuterVolumeSpecName: "config-out") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 08:57:57.679218 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.677489 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:57:57.685756 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.685731 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-web-config" (OuterVolumeSpecName: "web-config") pod "fcfde344-5055-4239-a97f-3a1f991d8ac0" (UID: "fcfde344-5055-4239-a97f-3a1f991d8ac0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 08:57:57.773465 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773350 2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-metrics-client-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773465 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773388 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-web-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773465 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773406 2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-kube-rbac-proxy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773465 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773418 2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-config\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773465 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773431 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-tls-assets\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773465 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773445 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773465 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773459 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773465 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773473 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-metrics-client-ca\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773871 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773484 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcfde344-5055-4239-a97f-3a1f991d8ac0-config-out\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773871 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773496 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773871 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773510 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773871 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773521 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773871 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773533 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcfde344-5055-4239-a97f-3a1f991d8ac0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773871 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773544 2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-secret-grpc-tls\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773871 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773555 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n76wn\" (UniqueName: \"kubernetes.io/projected/fcfde344-5055-4239-a97f-3a1f991d8ac0-kube-api-access-n76wn\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:57.773871 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:57.773567 2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fcfde344-5055-4239-a97f-3a1f991d8ac0-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 08:57:58.114248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114193 2574 generic.go:356] "Generic (PLEG): container finished" podID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerID="09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" exitCode=0 Jul 16 08:57:58.114248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114240 2574 generic.go:356] "Generic (PLEG): container finished" podID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerID="c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" exitCode=0 Jul 16 08:57:58.114248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114246 2574 generic.go:356] "Generic (PLEG): container finished" podID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerID="49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" exitCode=0 Jul 16 08:57:58.114248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114252 2574 generic.go:356] "Generic (PLEG): container finished" podID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerID="a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" exitCode=0 Jul 16 08:57:58.114248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114259 2574 generic.go:356] "Generic (PLEG): container finished" podID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerID="8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" exitCode=0 Jul 16 08:57:58.114248 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114265 2574 generic.go:356] "Generic (PLEG): container finished" podID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerID="6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" exitCode=0 Jul 16 08:57:58.114622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerDied","Data":"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689"} Jul 16 08:57:58.114622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114322 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.114622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerDied","Data":"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545"} Jul 16 08:57:58.114622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114417 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerDied","Data":"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7"} Jul 16 08:57:58.114622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114436 2574 scope.go:117] "RemoveContainer" containerID="09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" Jul 16 08:57:58.114622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114458 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerDied","Data":"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60"} Jul 16 08:57:58.114622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerDied","Data":"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381"} Jul 16 08:57:58.114622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114542 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerDied","Data":"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972"} Jul 16 08:57:58.114622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.114582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fcfde344-5055-4239-a97f-3a1f991d8ac0","Type":"ContainerDied","Data":"c46e92d34ffbeb378af0643801a6bea87e4949f01656ec618a3d13713184d880"} Jul 16 08:57:58.123407 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.123384 2574 scope.go:117] "RemoveContainer" containerID="c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" Jul 16 08:57:58.130765 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.130744 2574 scope.go:117] "RemoveContainer" containerID="49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" Jul 16 08:57:58.138350 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.138321 2574 scope.go:117] "RemoveContainer" containerID="a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" Jul 16 08:57:58.141593 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.141566 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 08:57:58.145461 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.145432 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 08:57:58.148868 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.148848 2574 scope.go:117] "RemoveContainer" containerID="8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" Jul 16 08:57:58.156081 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.156055 2574 scope.go:117] "RemoveContainer" containerID="6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" Jul 16 08:57:58.164023 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.164006 2574 scope.go:117] "RemoveContainer" containerID="cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299" Jul 16 08:57:58.170906 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.170889 2574 scope.go:117] "RemoveContainer" containerID="09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" Jul 16 08:57:58.171210 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:57:58.171184 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": container with ID starting with 09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689 not found: ID does not exist" containerID="09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" Jul 16 08:57:58.171307 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.171218 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689"} err="failed to get container status \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": rpc error: code = NotFound desc = could not find container \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": container with ID starting with 09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689 not found: ID does not exist" Jul 16 08:57:58.171307 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.171256 2574 scope.go:117] "RemoveContainer" containerID="c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" Jul 16 08:57:58.171516 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:57:58.171500 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": container with ID starting with c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545 not found: ID does not exist" containerID="c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" Jul 16 08:57:58.171559 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.171521 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545"} err="failed to get container status \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": rpc error: code = NotFound desc = could not find container \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": container with ID starting with c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545 not found: ID does not exist" Jul 16 08:57:58.171559 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.171535 2574 scope.go:117] "RemoveContainer" containerID="49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" Jul 16 08:57:58.171756 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:57:58.171739 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": container with ID starting with 49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7 not found: ID does not exist" containerID="49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" Jul 16 08:57:58.171800 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.171759 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7"} err="failed to get container status \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": rpc error: code = NotFound desc = could not find container \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": container with ID starting with 49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7 not found: ID does not exist" Jul 16 08:57:58.171800 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.171772 2574 scope.go:117] "RemoveContainer" containerID="a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" Jul 16 08:57:58.171977 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:57:58.171960 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": container with ID starting with a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60 not found: ID does not exist" containerID="a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" Jul 16 08:57:58.172012 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.171982 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60"} err="failed to get container status \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": rpc error: code = NotFound desc = could not find container \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": container with ID starting with a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60 not found: ID does not exist" Jul 16 08:57:58.172012 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.171996 2574 scope.go:117] "RemoveContainer" containerID="8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" Jul 16 08:57:58.172296 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:57:58.172277 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": container with ID starting with 8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381 not found: ID does not exist" containerID="8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" Jul 16 08:57:58.172356 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.172302 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381"} err="failed to get container status \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": rpc error: code = NotFound desc = could not find container \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": container with ID starting with 8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381 not found: ID does not exist" Jul 16 08:57:58.172356 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.172319 2574 scope.go:117] "RemoveContainer" containerID="6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" Jul 16 08:57:58.172555 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:57:58.172537 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": container with ID starting with 6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972 not found: ID does not exist" containerID="6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" Jul 16 08:57:58.172599 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.172569 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972"} err="failed to get container status \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": rpc error: code = NotFound desc = could not find container \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": container with ID starting with 6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972 not found: ID does not exist" Jul 16 08:57:58.172599 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.172583 2574 scope.go:117] "RemoveContainer" containerID="cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299" Jul 16 08:57:58.172802 ip-10-0-128-59 kubenswrapper[2574]: E0716 08:57:58.172778 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": container with ID starting with cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299 not found: ID does not exist" containerID="cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299" Jul 16 08:57:58.172841 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.172799 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299"} err="failed to get container status \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": rpc error: code = NotFound desc = could not find container \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": container with ID starting with cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299 not found: ID does not exist" Jul 16 08:57:58.172841 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.172812 2574 scope.go:117] "RemoveContainer" containerID="09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" Jul 16 08:57:58.173032 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173014 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689"} err="failed to get container status \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": rpc error: code = NotFound desc = could not find container \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": container with ID starting with 09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689 not found: ID does not exist" Jul 16 08:57:58.173079 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173033 2574 scope.go:117] "RemoveContainer" containerID="c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" Jul 16 08:57:58.173259 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173235 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545"} err="failed to get container status \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": rpc error: code = NotFound desc = could not find container \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": container with ID starting with c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545 not found: ID does not exist" Jul 16 08:57:58.173259 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173259 2574 scope.go:117] "RemoveContainer" containerID="49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" Jul 16 08:57:58.173477 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173460 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7"} err="failed to get container status \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": rpc error: code = NotFound desc = could not find container \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": container with ID starting with 49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7 not found: ID does not exist" Jul 16 08:57:58.173525 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173477 2574 scope.go:117] "RemoveContainer" containerID="a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" Jul 16 08:57:58.173702 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173686 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60"} err="failed to get container status \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": rpc error: code = NotFound desc = could not find container \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": container with ID starting with a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60 not found: ID does not exist" Jul 16 08:57:58.173702 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173702 2574 scope.go:117] "RemoveContainer" containerID="8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" Jul 16 08:57:58.173913 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173896 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381"} err="failed to get container status \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": rpc error: code = NotFound desc = could not find container \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": container with ID starting with 8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381 not found: ID does not exist" Jul 16 08:57:58.173981 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.173915 2574 scope.go:117] "RemoveContainer" containerID="6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" Jul 16 08:57:58.174109 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.174093 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972"} err="failed to get container status \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": rpc error: code = NotFound desc = could not find container \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": container with ID starting with 6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972 not found: ID does not exist" Jul 16 08:57:58.174146 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.174110 2574 scope.go:117] "RemoveContainer" containerID="cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299" Jul 16 08:57:58.174336 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.174315 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299"} err="failed to get container status \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": rpc error: code = NotFound desc = could not find container \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": container with ID starting with cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299 not found: ID does not exist" Jul 16 08:57:58.174407 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.174337 2574 scope.go:117] "RemoveContainer" containerID="09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" Jul 16 08:57:58.174579 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.174543 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689"} err="failed to get container status \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": rpc error: code = NotFound desc = could not find container \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": container with ID starting with 09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689 not found: ID does not exist" Jul 16 08:57:58.174579 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.174565 2574 scope.go:117] "RemoveContainer" containerID="c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" Jul 16 08:57:58.174822 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.174795 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545"} err="failed to get container status \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": rpc error: code = NotFound desc = could not find container \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": container with ID starting with c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545 not found: ID does not exist" Jul 16 08:57:58.174822 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.174818 2574 scope.go:117] "RemoveContainer" containerID="49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" Jul 16 08:57:58.175208 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.175182 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7"} err="failed to get container status \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": rpc error: code = NotFound desc = could not find container \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": container with ID starting with 49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7 not found: ID does not exist" Jul 16 08:57:58.175208 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.175208 2574 scope.go:117] "RemoveContainer" containerID="a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" Jul 16 08:57:58.175496 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.175468 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60"} err="failed to get container status \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": rpc error: code = NotFound desc = could not find container \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": container with ID starting with a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60 not found: ID does not exist" Jul 16 08:57:58.175496 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.175496 2574 scope.go:117] "RemoveContainer" containerID="8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" Jul 16 08:57:58.175768 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.175744 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381"} err="failed to get container status \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": rpc error: code = NotFound desc = could not find container \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": container with ID starting with 8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381 not found: ID does not exist" Jul 16 08:57:58.175839 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.175770 2574 scope.go:117] "RemoveContainer" containerID="6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" Jul 16 08:57:58.176030 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.176007 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972"} err="failed to get container status \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": rpc error: code = NotFound desc = could not find container \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": container with ID starting with 6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972 not found: ID does not exist" Jul 16 08:57:58.176119 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.176032 2574 scope.go:117] "RemoveContainer" containerID="cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299" Jul 16 08:57:58.176339 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.176321 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299"} err="failed to get container status \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": rpc error: code = NotFound desc = could not find container \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": container with ID starting with cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299 not found: ID does not exist" Jul 16 08:57:58.176408 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.176340 2574 scope.go:117] "RemoveContainer" containerID="09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" Jul 16 08:57:58.176600 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.176575 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689"} err="failed to get container status \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": rpc error: code = NotFound desc = could not find container \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": container with ID starting with 09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689 not found: ID does not exist" Jul 16 08:57:58.176649 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.176604 2574 scope.go:117] "RemoveContainer" containerID="c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" Jul 16 08:57:58.176857 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.176834 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545"} err="failed to get container status \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": rpc error: code = NotFound desc = could not find container \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": container with ID starting with c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545 not found: ID does not exist" Jul 16 08:57:58.176932 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.176858 2574 scope.go:117] "RemoveContainer" containerID="49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" Jul 16 08:57:58.177107 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177088 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7"} err="failed to get container status \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": rpc error: code = NotFound desc = could not find container \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": container with ID starting with 49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7 not found: ID does not exist" Jul 16 08:57:58.177173 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177107 2574 scope.go:117] "RemoveContainer" containerID="a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" Jul 16 08:57:58.177315 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177298 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 08:57:58.177387 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177368 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60"} err="failed to get container status \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": rpc error: code = NotFound desc = could not find container \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": container with ID starting with a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60 not found: ID does not exist" Jul 16 08:57:58.177427 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177388 2574 scope.go:117] "RemoveContainer" containerID="8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" Jul 16 08:57:58.177607 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177590 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381"} err="failed to get container status \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": rpc error: code = NotFound desc = could not find container \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": container with ID starting with 8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381 not found: ID does not exist" Jul 16 08:57:58.177607 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177602 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="config-reloader" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177618 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="config-reloader" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177634 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="init-config-reloader" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177640 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="init-config-reloader" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177651 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="prometheus" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177656 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="prometheus" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177662 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy-web" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177668 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy-web" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177607 2574 scope.go:117] "RemoveContainer" containerID="6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177680 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="thanos-sidecar" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177687 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="thanos-sidecar" Jul 16 08:57:58.177695 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177699 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177709 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177718 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy-thanos" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177723 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy-thanos" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177773 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy-thanos" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177787 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy-web" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177797 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="prometheus" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177806 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="config-reloader" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177814 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="kube-rbac-proxy" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177824 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" containerName="thanos-sidecar" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177879 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972"} err="failed to get container status \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": rpc error: code = NotFound desc = could not find container \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": container with ID starting with 6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972 not found: ID does not exist" Jul 16 08:57:58.178097 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.177903 2574 scope.go:117] "RemoveContainer" containerID="cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299" Jul 16 08:57:58.178481 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.178134 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299"} err="failed to get container status \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": rpc error: code = NotFound desc = could not find container \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": container with ID starting with cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299 not found: ID does not exist" Jul 16 08:57:58.178481 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.178173 2574 scope.go:117] "RemoveContainer" containerID="09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" Jul 16 08:57:58.178481 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.178362 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689"} err="failed to get container status \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": rpc error: code = NotFound desc = could not find container \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": container with ID starting with 09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689 not found: ID does not exist" Jul 16 08:57:58.178481 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.178375 2574 scope.go:117] "RemoveContainer" containerID="c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" Jul 16 08:57:58.178622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.178569 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545"} err="failed to get container status \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": rpc error: code = NotFound desc = could not find container \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": container with ID starting with c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545 not found: ID does not exist" Jul 16 08:57:58.178622 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.178582 2574 scope.go:117] "RemoveContainer" containerID="49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" Jul 16 08:57:58.178806 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.178789 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7"} err="failed to get container status \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": rpc error: code = NotFound desc = could not find container \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": container with ID starting with 49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7 not found: ID does not exist" Jul 16 08:57:58.178852 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.178807 2574 scope.go:117] "RemoveContainer" containerID="a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" Jul 16 08:57:58.179036 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179015 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60"} err="failed to get container status \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": rpc error: code = NotFound desc = could not find container \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": container with ID starting with a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60 not found: ID does not exist" Jul 16 08:57:58.179080 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179039 2574 scope.go:117] "RemoveContainer" containerID="8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" Jul 16 08:57:58.179321 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179300 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381"} err="failed to get container status \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": rpc error: code = NotFound desc = could not find container \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": container with ID starting with 8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381 not found: ID does not exist" Jul 16 08:57:58.179321 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179320 2574 scope.go:117] "RemoveContainer" containerID="6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" Jul 16 08:57:58.179519 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179501 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972"} err="failed to get container status \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": rpc error: code = NotFound desc = could not find container \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": container with ID starting with 6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972 not found: ID does not exist" Jul 16 08:57:58.179565 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179519 2574 scope.go:117] "RemoveContainer" containerID="cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299" Jul 16 08:57:58.179701 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179687 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299"} err="failed to get container status \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": rpc error: code = NotFound desc = could not find container \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": container with ID starting with cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299 not found: ID does not exist" Jul 16 08:57:58.179701 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179701 2574 scope.go:117] "RemoveContainer" containerID="09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689" Jul 16 08:57:58.179887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179872 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689"} err="failed to get container status \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": rpc error: code = NotFound desc = could not find container \"09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689\": container with ID starting with 09965793ee184be64cbec945189837e0d9bce191eafc5f0440ac5a534ec51689 not found: ID does not exist" Jul 16 08:57:58.179887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.179886 2574 scope.go:117] "RemoveContainer" containerID="c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545" Jul 16 08:57:58.180047 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180033 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545"} err="failed to get container status \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": rpc error: code = NotFound desc = could not find container \"c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545\": container with ID starting with c3963d8173e81d3940ef99590baae4132617551c50f6f5c47c5418d8a4a7b545 not found: ID does not exist" Jul 16 08:57:58.180047 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180046 2574 scope.go:117] "RemoveContainer" containerID="49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7" Jul 16 08:57:58.180259 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180237 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7"} err="failed to get container status \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": rpc error: code = NotFound desc = could not find container \"49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7\": container with ID starting with 49ce008ba0f73d8b079c7e02a3405066e126c06411c9dcb10778e2f64c7523f7 not found: ID does not exist" Jul 16 08:57:58.180422 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180260 2574 scope.go:117] "RemoveContainer" containerID="a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60" Jul 16 08:57:58.180493 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180453 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60"} err="failed to get container status \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": rpc error: code = NotFound desc = could not find container \"a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60\": container with ID starting with a88d8269e00541745db992225fc63378cc4255e4ade2cc2c12a3d0ca18be2a60 not found: ID does not exist" Jul 16 08:57:58.180493 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180468 2574 scope.go:117] "RemoveContainer" containerID="8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381" Jul 16 08:57:58.180684 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180666 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381"} err="failed to get container status \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": rpc error: code = NotFound desc = could not find container \"8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381\": container with ID starting with 8158c89502e257beaabb5ffb9672adf56e7941519fc3a77aab4f8a05476eb381 not found: ID does not exist" Jul 16 08:57:58.180744 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180685 2574 scope.go:117] "RemoveContainer" containerID="6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972" Jul 16 08:57:58.180872 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180855 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972"} err="failed to get container status \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": rpc error: code = NotFound desc = could not find container \"6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972\": container with ID starting with 6f70d4140f735b449d6f44e257f9f7ab695d7dd7439cf34fdb4fd4be50d1f972 not found: ID does not exist" Jul 16 08:57:58.180909 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.180873 2574 scope.go:117] "RemoveContainer" containerID="cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299" Jul 16 08:57:58.181058 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.181042 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299"} err="failed to get container status \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": rpc error: code = NotFound desc = could not find container \"cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299\": container with ID starting with cb21e3ce44dbb697ddb2fffab2a60fb08163324423776ef46345303826218299 not found: ID does not exist" Jul 16 08:57:58.183493 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.183479 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.186050 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186030 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 08:57:58.186186 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186083 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 08:57:58.186285 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186245 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vzz9p\"" Jul 16 08:57:58.186333 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186313 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 08:57:58.186388 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186375 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 08:57:58.186388 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186380 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 08:57:58.186523 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186316 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2rrv8aqba1824\"" Jul 16 08:57:58.186523 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186315 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 08:57:58.186523 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186502 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 08:57:58.186707 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186678 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 08:57:58.186707 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186684 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 08:57:58.186875 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186708 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 08:57:58.186875 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186687 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 08:57:58.186875 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186801 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 08:57:58.187071 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.186914 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 08:57:58.189611 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.189592 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 08:57:58.191789 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.191750 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 08:57:58.199090 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.199065 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 08:57:58.276811 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.276767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.276811 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.276809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.276830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.276849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-web-config\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.276927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.276974 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277011 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzmqf\" (UniqueName: \"kubernetes.io/projected/46e348c6-6e0a-4346-8997-0f1273b4abde-kube-api-access-wzmqf\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277264 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277282 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46e348c6-6e0a-4346-8997-0f1273b4abde-config-out\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277338 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277717 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277352 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277717 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46e348c6-6e0a-4346-8997-0f1273b4abde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277717 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277717 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.277717 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.277449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-config\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378181 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46e348c6-6e0a-4346-8997-0f1273b4abde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378181 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378181 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378504 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-config\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378504 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378332 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378504 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378504 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378504 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-web-config\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378504 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378504 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378504 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378555 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzmqf\" (UniqueName: \"kubernetes.io/projected/46e348c6-6e0a-4346-8997-0f1273b4abde-kube-api-access-wzmqf\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46e348c6-6e0a-4346-8997-0f1273b4abde-config-out\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.378887 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378798 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.379321 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.378962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.379321 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.379039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.379321 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.379088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.381272 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.380580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.381415 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.381389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.382625 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.381717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.382625 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.381964 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.382625 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.382260 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-web-config\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.382625 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.382529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.382625 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.382486 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.382969 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.382727 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.383122 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.383068 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.383782 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.383393 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-config\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.383782 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.383473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46e348c6-6e0a-4346-8997-0f1273b4abde-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.383782 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.383589 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.383928 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.383854 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.384313 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.384294 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/46e348c6-6e0a-4346-8997-0f1273b4abde-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.384776 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.384759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46e348c6-6e0a-4346-8997-0f1273b4abde-config-out\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.385223 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.385206 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/46e348c6-6e0a-4346-8997-0f1273b4abde-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.391493 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.391470 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzmqf\" (UniqueName: \"kubernetes.io/projected/46e348c6-6e0a-4346-8997-0f1273b4abde-kube-api-access-wzmqf\") pod \"prometheus-k8s-0\" (UID: \"46e348c6-6e0a-4346-8997-0f1273b4abde\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.494721 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.494674 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:57:58.636414 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:58.636315 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 08:57:58.639393 ip-10-0-128-59 kubenswrapper[2574]: W0716 08:57:58.639360 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e348c6_6e0a_4346_8997_0f1273b4abde.slice/crio-a8f648295d6dd74ef0352e5e4d85c7e51cd9759da1068de30f91506456398e4f WatchSource:0}: Error finding container a8f648295d6dd74ef0352e5e4d85c7e51cd9759da1068de30f91506456398e4f: Status 404 returned error can't find the container with id a8f648295d6dd74ef0352e5e4d85c7e51cd9759da1068de30f91506456398e4f Jul 16 08:57:59.054826 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:59.054737 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcfde344-5055-4239-a97f-3a1f991d8ac0" path="/var/lib/kubelet/pods/fcfde344-5055-4239-a97f-3a1f991d8ac0/volumes" Jul 16 08:57:59.120796 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:59.120758 2574 generic.go:356] "Generic (PLEG): container finished" podID="46e348c6-6e0a-4346-8997-0f1273b4abde" containerID="eb497cbeb3a56dc02306eb38a3f71c33d47eb42a3cafc7436ab42fc3dc1f50b6" exitCode=0 Jul 16 08:57:59.120993 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:59.120826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46e348c6-6e0a-4346-8997-0f1273b4abde","Type":"ContainerDied","Data":"eb497cbeb3a56dc02306eb38a3f71c33d47eb42a3cafc7436ab42fc3dc1f50b6"} Jul 16 08:57:59.120993 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:57:59.120881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46e348c6-6e0a-4346-8997-0f1273b4abde","Type":"ContainerStarted","Data":"a8f648295d6dd74ef0352e5e4d85c7e51cd9759da1068de30f91506456398e4f"} Jul 16 08:58:00.128734 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:00.128691 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46e348c6-6e0a-4346-8997-0f1273b4abde","Type":"ContainerStarted","Data":"21a2f3f457d8e24973cdf912e39e61faf337bfd8edd60cdac53fe9547b17749d"} Jul 16 08:58:00.129201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:00.128801 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46e348c6-6e0a-4346-8997-0f1273b4abde","Type":"ContainerStarted","Data":"36b0c6b3ca1dd0cb9b3d74983b4749cbbc916eaef79ec99079a99d7432bcbcff"} Jul 16 08:58:00.129201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:00.128835 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46e348c6-6e0a-4346-8997-0f1273b4abde","Type":"ContainerStarted","Data":"10bab110c8580ac96e6bd009291ef86b15e7a0cb33de8db4a159ce0430998a2b"} Jul 16 08:58:00.129201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:00.128883 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46e348c6-6e0a-4346-8997-0f1273b4abde","Type":"ContainerStarted","Data":"3ea63444240d44a094d8417fa4012f4e5410b71c4c777855c4972b639f1a7f29"} Jul 16 08:58:00.129201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:00.128910 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46e348c6-6e0a-4346-8997-0f1273b4abde","Type":"ContainerStarted","Data":"9ada5371314ad57922093b19cdc0b3dc295d3a066fc2a9214505db0dc1b0e072"} Jul 16 08:58:00.129201 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:00.128943 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"46e348c6-6e0a-4346-8997-0f1273b4abde","Type":"ContainerStarted","Data":"f52fc9fede0a6bc702b2a74605137f116f8d2161c37e384c64cd12c50bf3e05d"} Jul 16 08:58:00.156062 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:00.155991 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.133251809 podStartE2EDuration="2.155966748s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.714939ms" isStatefulPod=false podCreationTimestamp="2026-07-16 08:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 08:58:00.154490798 +0000 UTC m=+193.528625089" watchObservedRunningTime="2026-07-16 08:58:00.155966748 +0000 UTC m=+193.530101037" Jul 16 08:58:03.495226 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:03.495179 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:58:58.495496 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:58.495459 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:58:58.512280 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:58.512248 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:58:59.378418 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:58:59.378385 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 08:59:46.970040 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:59:46.970007 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 08:59:46.970691 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:59:46.970458 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 08:59:46.976018 ip-10-0-128-59 kubenswrapper[2574]: I0716 08:59:46.975991 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 09:01:27.953859 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:27.953823 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79"] Jul 16 09:01:27.957297 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:27.957267 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:27.961760 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:27.961733 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 09:01:27.961909 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:27.961872 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hgv5l\"" Jul 16 09:01:27.961979 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:27.961921 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 09:01:27.961979 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:27.961909 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 09:01:27.962088 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:27.961986 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 09:01:27.987415 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:27.987376 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79"] Jul 16 09:01:28.017092 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.017051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8b8fd9d-aebe-4607-b178-df74a27c1d23-manifests\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.017092 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.017089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb6bl\" (UniqueName: \"kubernetes.io/projected/d8b8fd9d-aebe-4607-b178-df74a27c1d23-kube-api-access-kb6bl\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.017354 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.017169 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8b8fd9d-aebe-4607-b178-df74a27c1d23-tmp\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.017354 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.017196 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8b8fd9d-aebe-4607-b178-df74a27c1d23-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.017354 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.017225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8b8fd9d-aebe-4607-b178-df74a27c1d23-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.118060 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.118023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8b8fd9d-aebe-4607-b178-df74a27c1d23-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.118334 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.118258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8b8fd9d-aebe-4607-b178-df74a27c1d23-manifests\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.118334 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.118300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb6bl\" (UniqueName: \"kubernetes.io/projected/d8b8fd9d-aebe-4607-b178-df74a27c1d23-kube-api-access-kb6bl\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.118458 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.118355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8b8fd9d-aebe-4607-b178-df74a27c1d23-tmp\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.118458 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.118385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8b8fd9d-aebe-4607-b178-df74a27c1d23-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.118764 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.118731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d8b8fd9d-aebe-4607-b178-df74a27c1d23-manifests\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.120807 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.120772 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8b8fd9d-aebe-4607-b178-df74a27c1d23-tmp\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.121048 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.121028 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d8b8fd9d-aebe-4607-b178-df74a27c1d23-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.121109 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.121030 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8b8fd9d-aebe-4607-b178-df74a27c1d23-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.127542 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.127511 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb6bl\" (UniqueName: \"kubernetes.io/projected/d8b8fd9d-aebe-4607-b178-df74a27c1d23-kube-api-access-kb6bl\") pod \"opendatahub-operator-controller-manager-5b875787f8-78b79\" (UID: \"d8b8fd9d-aebe-4607-b178-df74a27c1d23\") " pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.267899 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.267791 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:28.405739 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.405709 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79"] Jul 16 09:01:28.408841 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:01:28.408804 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b8fd9d_aebe_4607_b178_df74a27c1d23.slice/crio-20cc7f0d94122efb949826629f0984c5b4644f76902e1b2a6af0ca99cc77ad2e WatchSource:0}: Error finding container 20cc7f0d94122efb949826629f0984c5b4644f76902e1b2a6af0ca99cc77ad2e: Status 404 returned error can't find the container with id 20cc7f0d94122efb949826629f0984c5b4644f76902e1b2a6af0ca99cc77ad2e Jul 16 09:01:28.410777 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.410755 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 09:01:28.949773 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:28.949732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" event={"ID":"d8b8fd9d-aebe-4607-b178-df74a27c1d23","Type":"ContainerStarted","Data":"20cc7f0d94122efb949826629f0984c5b4644f76902e1b2a6af0ca99cc77ad2e"} Jul 16 09:01:32.970535 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:32.970491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" event={"ID":"d8b8fd9d-aebe-4607-b178-df74a27c1d23","Type":"ContainerStarted","Data":"aed47320ab4983c2071276a29dd94388fcd70d6da477737ad6738c744b8d9388"} Jul 16 09:01:33.976756 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:33.976716 2574 generic.go:356] "Generic (PLEG): container finished" podID="d8b8fd9d-aebe-4607-b178-df74a27c1d23" containerID="aed47320ab4983c2071276a29dd94388fcd70d6da477737ad6738c744b8d9388" exitCode=0 Jul 16 09:01:33.977146 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:33.976803 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" event={"ID":"d8b8fd9d-aebe-4607-b178-df74a27c1d23","Type":"ContainerDied","Data":"aed47320ab4983c2071276a29dd94388fcd70d6da477737ad6738c744b8d9388"} Jul 16 09:01:34.983110 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:34.983064 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" event={"ID":"d8b8fd9d-aebe-4607-b178-df74a27c1d23","Type":"ContainerStarted","Data":"2d05e37673ca27329be5f5ae14e75f7c88cf5e13c8bbf1cb955532dd1bdb9afb"} Jul 16 09:01:34.983545 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:34.983247 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:35.004763 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:35.004507 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" podStartSLOduration=2.7008788409999998 podStartE2EDuration="8.004483422s" totalImagesPullingTime="4.152325292s" totalInitContainerRuntime="1.151279289s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:01:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:01:35.002951049 +0000 UTC m=+408.377085339" watchObservedRunningTime="2026-07-16 09:01:35.004483422 +0000 UTC m=+408.378617714" Jul 16 09:01:45.990082 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:45.990045 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5b875787f8-78b79" Jul 16 09:01:49.583381 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.583343 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-856d68f954-94qtz"] Jul 16 09:01:49.593738 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.593707 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:49.596311 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.596282 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 09:01:49.596476 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.596324 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4xhvr\"" Jul 16 09:01:49.597363 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.597336 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 09:01:49.597526 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.597390 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 09:01:49.597589 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.597528 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 09:01:49.599392 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.599366 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-856d68f954-94qtz"] Jul 16 09:01:49.707323 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.707286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9crc\" (UniqueName: \"kubernetes.io/projected/87d37615-95c8-4a58-86bd-6c7ad1410b29-kube-api-access-g9crc\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:49.707511 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.707350 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d37615-95c8-4a58-86bd-6c7ad1410b29-tmp\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:49.707511 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.707417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d37615-95c8-4a58-86bd-6c7ad1410b29-tls-certs\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:49.807953 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.807901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d37615-95c8-4a58-86bd-6c7ad1410b29-tmp\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:49.807953 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.807956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d37615-95c8-4a58-86bd-6c7ad1410b29-tls-certs\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:49.808297 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.808042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9crc\" (UniqueName: \"kubernetes.io/projected/87d37615-95c8-4a58-86bd-6c7ad1410b29-kube-api-access-g9crc\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:49.808297 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:01:49.808074 2574 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 16 09:01:49.808297 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:01:49.808187 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87d37615-95c8-4a58-86bd-6c7ad1410b29-tls-certs podName:87d37615-95c8-4a58-86bd-6c7ad1410b29 nodeName:}" failed. No retries permitted until 2026-07-16 09:01:50.308141988 +0000 UTC m=+423.682276257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/87d37615-95c8-4a58-86bd-6c7ad1410b29-tls-certs") pod "kube-auth-proxy-856d68f954-94qtz" (UID: "87d37615-95c8-4a58-86bd-6c7ad1410b29") : secret "kube-auth-proxy-tls" not found Jul 16 09:01:49.810422 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.810392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d37615-95c8-4a58-86bd-6c7ad1410b29-tmp\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:49.817450 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:49.817418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9crc\" (UniqueName: \"kubernetes.io/projected/87d37615-95c8-4a58-86bd-6c7ad1410b29-kube-api-access-g9crc\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:50.311886 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:50.311841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d37615-95c8-4a58-86bd-6c7ad1410b29-tls-certs\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:50.314631 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:50.314604 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d37615-95c8-4a58-86bd-6c7ad1410b29-tls-certs\") pod \"kube-auth-proxy-856d68f954-94qtz\" (UID: \"87d37615-95c8-4a58-86bd-6c7ad1410b29\") " pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:50.504853 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:50.504819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" Jul 16 09:01:50.646746 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:50.646596 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-856d68f954-94qtz"] Jul 16 09:01:50.649506 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:01:50.649470 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d37615_95c8_4a58_86bd_6c7ad1410b29.slice/crio-a6e3318e2871eb85a2640ce8f78758ea60a7a04d1419e54bbf26db25a9927070 WatchSource:0}: Error finding container a6e3318e2871eb85a2640ce8f78758ea60a7a04d1419e54bbf26db25a9927070: Status 404 returned error can't find the container with id a6e3318e2871eb85a2640ce8f78758ea60a7a04d1419e54bbf26db25a9927070 Jul 16 09:01:51.048932 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:51.048887 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" event={"ID":"87d37615-95c8-4a58-86bd-6c7ad1410b29","Type":"ContainerStarted","Data":"a6e3318e2871eb85a2640ce8f78758ea60a7a04d1419e54bbf26db25a9927070"} Jul 16 09:01:56.071646 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:56.071602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" event={"ID":"87d37615-95c8-4a58-86bd-6c7ad1410b29","Type":"ContainerStarted","Data":"1406425f8b9d7d5538d10327f96ca4c8cae234191935fc99a5048367c668ccd4"} Jul 16 09:01:56.090344 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:56.090291 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-856d68f954-94qtz" podStartSLOduration=2.706023959 podStartE2EDuration="7.090270933s" totalImagesPullingTime="4.384246974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:01:56.089735094 +0000 UTC m=+429.463869397" watchObservedRunningTime="2026-07-16 09:01:56.090270933 +0000 UTC m=+429.464405324" Jul 16 09:01:58.803413 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.803369 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv"] Jul 16 09:01:58.806822 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.806797 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.813302 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.813272 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 09:01:58.813811 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.813487 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-2qx2z\"" Jul 16 09:01:58.838455 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.838416 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv"] Jul 16 09:01:58.891940 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.891897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.892143 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.891959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.892143 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.891998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.892143 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.892074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.892143 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.892115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44wtg\" (UniqueName: \"kubernetes.io/projected/977280d7-4a3a-4fcb-87c4-c863fb79626a-kube-api-access-44wtg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.892143 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.892137 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977280d7-4a3a-4fcb-87c4-c863fb79626a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.892353 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.892187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.892353 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.892239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.892353 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.892280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.993705 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.993659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.993705 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.993713 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44wtg\" (UniqueName: \"kubernetes.io/projected/977280d7-4a3a-4fcb-87c4-c863fb79626a-kube-api-access-44wtg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.993738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977280d7-4a3a-4fcb-87c4-c863fb79626a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.993762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.993795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.993845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.993881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.993931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.993983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994393 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.994288 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994542 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.994457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994664 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.994540 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.994755 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.994724 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.995100 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.995079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977280d7-4a3a-4fcb-87c4-c863fb79626a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.996641 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.996618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:58.996857 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:58.996839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:59.017305 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.017272 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44wtg\" (UniqueName: \"kubernetes.io/projected/977280d7-4a3a-4fcb-87c4-c863fb79626a-kube-api-access-44wtg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:59.022316 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.022292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977280d7-4a3a-4fcb-87c4-c863fb79626a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4f9phv\" (UID: \"977280d7-4a3a-4fcb-87c4-c863fb79626a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:59.117548 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.117514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:01:59.220554 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.220508 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc"] Jul 16 09:01:59.224528 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.224374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.227674 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.227442 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 09:01:59.227674 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.227456 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-m29c5\"" Jul 16 09:01:59.227674 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.227487 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 09:01:59.227674 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.227549 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 09:01:59.227674 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.227562 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 09:01:59.227674 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.227443 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 09:01:59.237069 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.237043 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc"] Jul 16 09:01:59.276168 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.276128 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv"] Jul 16 09:01:59.278432 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:01:59.278400 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977280d7_4a3a_4fcb_87c4_c863fb79626a.slice/crio-911b75a5d64f46c40f21692a78c962b460b04f3889f0c47b90f223d50893d5eb WatchSource:0}: Error finding container 911b75a5d64f46c40f21692a78c962b460b04f3889f0c47b90f223d50893d5eb: Status 404 returned error can't find the container with id 911b75a5d64f46c40f21692a78c962b460b04f3889f0c47b90f223d50893d5eb Jul 16 09:01:59.396590 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.396468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a516d268-427a-400a-8911-9625fe05627f-cert\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.396771 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.396600 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a516d268-427a-400a-8911-9625fe05627f-manager-config\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.396771 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.396638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a516d268-427a-400a-8911-9625fe05627f-metrics-cert\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.396771 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.396692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrvp\" (UniqueName: \"kubernetes.io/projected/a516d268-427a-400a-8911-9625fe05627f-kube-api-access-qwrvp\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.497281 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.497222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrvp\" (UniqueName: \"kubernetes.io/projected/a516d268-427a-400a-8911-9625fe05627f-kube-api-access-qwrvp\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.497554 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.497326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a516d268-427a-400a-8911-9625fe05627f-cert\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.497554 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.497400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a516d268-427a-400a-8911-9625fe05627f-manager-config\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.497554 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.497419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a516d268-427a-400a-8911-9625fe05627f-metrics-cert\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.498331 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.498308 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a516d268-427a-400a-8911-9625fe05627f-manager-config\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.500055 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.500026 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a516d268-427a-400a-8911-9625fe05627f-cert\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.500190 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.500126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a516d268-427a-400a-8911-9625fe05627f-metrics-cert\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.508687 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.508663 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrvp\" (UniqueName: \"kubernetes.io/projected/a516d268-427a-400a-8911-9625fe05627f-kube-api-access-qwrvp\") pod \"lws-controller-manager-74556d96d9-xgdgc\" (UID: \"a516d268-427a-400a-8911-9625fe05627f\") " pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.536530 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.536496 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:01:59.703868 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:01:59.703839 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc"] Jul 16 09:01:59.706355 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:01:59.706320 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda516d268_427a_400a_8911_9625fe05627f.slice/crio-9186ab65918bf62b94cfbfeddecaa2508e5a1a44c4cb772b7aade3ce620b95ec WatchSource:0}: Error finding container 9186ab65918bf62b94cfbfeddecaa2508e5a1a44c4cb772b7aade3ce620b95ec: Status 404 returned error can't find the container with id 9186ab65918bf62b94cfbfeddecaa2508e5a1a44c4cb772b7aade3ce620b95ec Jul 16 09:02:00.089369 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:00.089267 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" event={"ID":"977280d7-4a3a-4fcb-87c4-c863fb79626a","Type":"ContainerStarted","Data":"911b75a5d64f46c40f21692a78c962b460b04f3889f0c47b90f223d50893d5eb"} Jul 16 09:02:00.090641 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:00.090612 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" event={"ID":"a516d268-427a-400a-8911-9625fe05627f","Type":"ContainerStarted","Data":"9186ab65918bf62b94cfbfeddecaa2508e5a1a44c4cb772b7aade3ce620b95ec"} Jul 16 09:02:03.546259 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:03.546215 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:02:03.546583 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:03.546299 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:02:03.546583 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:03.546354 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:02:04.114413 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:04.114364 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" event={"ID":"a516d268-427a-400a-8911-9625fe05627f","Type":"ContainerStarted","Data":"f7ca94265d9f5da7b4be76f6f42a7df7cafee7eab694699c8494d26eb022981e"} Jul 16 09:02:04.114656 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:04.114436 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:02:04.116068 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:04.116029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" event={"ID":"977280d7-4a3a-4fcb-87c4-c863fb79626a","Type":"ContainerStarted","Data":"14666bacb27d6fa5cf1391916ef549e6ebf89240e54c2e8d0207b79b1f5be09a"} Jul 16 09:02:04.118438 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:04.118421 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:02:04.123410 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:04.123389 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:02:04.142354 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:04.142299 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" podStartSLOduration=1.307553919 podStartE2EDuration="5.142281795s" totalImagesPullingTime="3.834727876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:02:04.141444546 +0000 UTC m=+437.515578836" watchObservedRunningTime="2026-07-16 09:02:04.142281795 +0000 UTC m=+437.516416085" Jul 16 09:02:04.162592 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:04.162533 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" podStartSLOduration=1.897959646 podStartE2EDuration="6.162515672s" totalImagesPullingTime="4.264556026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:02:04.160894072 +0000 UTC m=+437.535028362" watchObservedRunningTime="2026-07-16 09:02:04.162515672 +0000 UTC m=+437.536649962" Jul 16 09:02:05.120873 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:05.120830 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:02:05.121802 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:05.121784 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4f9phv" Jul 16 09:02:15.124070 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:02:15.123979 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74556d96d9-xgdgc" Jul 16 09:04:40.640097 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.640060 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:04:40.643389 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.643369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:04:40.645751 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.645724 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 09:04:40.645886 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.645772 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 09:04:40.646650 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.646626 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6dz8w\"" Jul 16 09:04:40.646763 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.646658 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 09:04:40.652427 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.652403 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:04:40.689182 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.689146 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:04:40.770586 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.770561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73e74bc7-8621-4741-ae0e-e325fe735ec4-config-file\") pod \"limitador-limitador-54d6d988d5-jl6gr\" (UID: \"73e74bc7-8621-4741-ae0e-e325fe735ec4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:04:40.770690 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.770593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwrn\" (UniqueName: \"kubernetes.io/projected/73e74bc7-8621-4741-ae0e-e325fe735ec4-kube-api-access-2jwrn\") pod \"limitador-limitador-54d6d988d5-jl6gr\" (UID: \"73e74bc7-8621-4741-ae0e-e325fe735ec4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:04:40.871096 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.871068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73e74bc7-8621-4741-ae0e-e325fe735ec4-config-file\") pod \"limitador-limitador-54d6d988d5-jl6gr\" (UID: \"73e74bc7-8621-4741-ae0e-e325fe735ec4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:04:40.871226 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.871098 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwrn\" (UniqueName: \"kubernetes.io/projected/73e74bc7-8621-4741-ae0e-e325fe735ec4-kube-api-access-2jwrn\") pod \"limitador-limitador-54d6d988d5-jl6gr\" (UID: \"73e74bc7-8621-4741-ae0e-e325fe735ec4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:04:40.871713 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.871693 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/73e74bc7-8621-4741-ae0e-e325fe735ec4-config-file\") pod \"limitador-limitador-54d6d988d5-jl6gr\" (UID: \"73e74bc7-8621-4741-ae0e-e325fe735ec4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:04:40.880666 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.880643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwrn\" (UniqueName: \"kubernetes.io/projected/73e74bc7-8621-4741-ae0e-e325fe735ec4-kube-api-access-2jwrn\") pod \"limitador-limitador-54d6d988d5-jl6gr\" (UID: \"73e74bc7-8621-4741-ae0e-e325fe735ec4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:04:40.955847 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:40.955781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:04:41.083508 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:41.083478 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:04:41.086416 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:04:41.086390 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e74bc7_8621_4741_ae0e_e325fe735ec4.slice/crio-3464b803802f913f448953622a73df1f187c52b38026d788e0f028bb54c41782 WatchSource:0}: Error finding container 3464b803802f913f448953622a73df1f187c52b38026d788e0f028bb54c41782: Status 404 returned error can't find the container with id 3464b803802f913f448953622a73df1f187c52b38026d788e0f028bb54c41782 Jul 16 09:04:41.814092 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:41.814053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" event={"ID":"73e74bc7-8621-4741-ae0e-e325fe735ec4","Type":"ContainerStarted","Data":"3464b803802f913f448953622a73df1f187c52b38026d788e0f028bb54c41782"} Jul 16 09:04:43.824214 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:43.824184 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" event={"ID":"73e74bc7-8621-4741-ae0e-e325fe735ec4","Type":"ContainerStarted","Data":"2e1e3fbf43c778c5a8e260e3fe3931d2e7529206a33366aad0265ddff1a4316a"} Jul 16 09:04:43.824503 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:43.824249 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:04:43.846217 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:43.846174 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" podStartSLOduration=1.234598471 podStartE2EDuration="3.846136392s" totalImagesPullingTime="2.611537921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:04:43.845324306 +0000 UTC m=+597.219458593" watchObservedRunningTime="2026-07-16 09:04:43.846136392 +0000 UTC m=+597.220270681" Jul 16 09:04:47.001852 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:47.001820 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:04:47.002312 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:47.002204 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:04:54.830366 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:04:54.830337 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-jl6gr" Jul 16 09:05:54.369347 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:05:54.369308 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:06:19.326172 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:06:19.326118 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:06:32.131410 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:06:32.131369 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:06:37.020044 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:06:37.019949 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:06:46.821173 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:06:46.821113 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:07:03.911705 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:03.911661 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:07:13.459260 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.459226 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j"] Jul 16 09:07:13.462363 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.462342 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.465933 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.465904 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 09:07:13.465933 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.465927 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jn4fs\"" Jul 16 09:07:13.466143 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.465944 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 09:07:13.466143 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.465982 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 09:07:13.475659 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.475634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j"] Jul 16 09:07:13.544324 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.544294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.544324 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.544323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.544497 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.544363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.544497 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.544419 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.544497 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.544488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwx47\" (UniqueName: \"kubernetes.io/projected/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-kube-api-access-mwx47\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.544647 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.544556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.645708 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.645681 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.645823 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.645720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwx47\" (UniqueName: \"kubernetes.io/projected/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-kube-api-access-mwx47\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.645823 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.645753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.645823 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.645775 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.645823 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.645793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.645823 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.645825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.646148 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.646127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.646148 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.646139 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.646261 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.646223 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.648113 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.648087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.648210 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.648196 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.656423 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.656403 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwx47\" (UniqueName: \"kubernetes.io/projected/83a87d6d-fc5f-4128-bdd6-1f7823c38f1f-kube-api-access-mwx47\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j\" (UID: \"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.772046 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.772000 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:13.895472 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.895451 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j"] Jul 16 09:07:13.897767 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:07:13.897737 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83a87d6d_fc5f_4128_bdd6_1f7823c38f1f.slice/crio-26f9ae2ef0ce772b8a8eb17d692ce4f4fae9e445aaa0aa3ecda85a1f3437861f WatchSource:0}: Error finding container 26f9ae2ef0ce772b8a8eb17d692ce4f4fae9e445aaa0aa3ecda85a1f3437861f: Status 404 returned error can't find the container with id 26f9ae2ef0ce772b8a8eb17d692ce4f4fae9e445aaa0aa3ecda85a1f3437861f Jul 16 09:07:13.899137 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:13.899120 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 09:07:14.475616 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:14.475511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" event={"ID":"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f","Type":"ContainerStarted","Data":"26f9ae2ef0ce772b8a8eb17d692ce4f4fae9e445aaa0aa3ecda85a1f3437861f"} Jul 16 09:07:15.480963 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:15.480924 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" event={"ID":"83a87d6d-fc5f-4128-bdd6-1f7823c38f1f","Type":"ContainerStarted","Data":"737a01ea8eb8be4e69b8b0d40b21be41c5ff3a4bdc6327a6f91d5670d2db44e5"} Jul 16 09:07:15.498126 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:15.498086 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" podStartSLOduration=1.413382223 podStartE2EDuration="2.498070964s" totalImagesPullingTime="1.084688741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:07:15.497242417 +0000 UTC m=+748.871376706" watchObservedRunningTime="2026-07-16 09:07:15.498070964 +0000 UTC m=+748.872205254" Jul 16 09:07:16.032493 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:16.032461 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:07:23.772828 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:23.772788 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:23.773392 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:23.772844 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:23.786086 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:23.786059 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:07:24.536306 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:07:24.536277 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j" Jul 16 09:08:39.813708 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:08:39.813678 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:08:52.516330 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:08:52.516289 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:09:01.506728 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:09:01.506689 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:09:10.407894 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:09:10.407854 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:09:20.812639 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:09:20.812593 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:09:47.033353 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:09:47.028018 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:09:47.036087 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:09:47.036051 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:09:49.524437 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:09:49.524399 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:10:03.924176 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:10:03.924121 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:10:39.807543 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:10:39.807499 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:10:47.326937 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:10:47.326897 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:10:58.224845 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:10:58.224806 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:11:07.027687 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:11:07.027585 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:11:18.230572 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:11:18.230529 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:11:28.914959 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:11:28.914921 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:11:40.929478 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:11:40.929433 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:11:50.913048 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:11:50.913012 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:12:12.939007 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:12:12.938964 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:12:22.144364 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:12:22.144321 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:12:45.423881 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:12:45.423791 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:13:00.925416 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:13:00.925374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:13:12.823114 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:13:12.823078 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:13:22.224411 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:13:22.224378 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:13:32.326235 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:13:32.326197 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:13:42.118491 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:13:42.118447 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:13:51.621192 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:13:51.621137 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:14:47.061681 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:14:47.061652 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:14:47.064468 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:14:47.064447 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:15:41.110082 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:15:41.109994 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:15:50.212716 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:15:50.212678 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:16:03.210062 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:16:03.210017 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:16:17.322406 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:16:17.322368 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:16:27.122992 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:16:27.122958 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:16:37.723394 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:16:37.723354 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:16:46.713411 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:16:46.713377 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:16:57.613396 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:16:57.613354 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:17:07.408758 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:17:07.408678 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:17:18.739715 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:17:18.739675 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:17:30.223173 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:17:30.223109 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:17:52.420654 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:17:52.420612 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:18:01.726364 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:18:01.726323 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:18:12.725232 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:18:12.725194 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:18:25.020651 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:18:25.020610 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:18:43.614345 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:18:43.614256 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:18:55.215350 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:18:55.215305 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:19:04.617095 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:19:04.617057 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:19:19.125400 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:19:19.125365 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:19:28.608735 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:19:28.608689 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:19:41.225000 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:19:41.224956 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:19:47.088240 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:19:47.088201 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:19:47.091582 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:19:47.091562 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:19:50.718186 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:19:50.718127 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:20:00.618796 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:20:00.618754 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:20:13.521785 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:20:13.521698 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:20:22.910563 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:20:22.910518 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:20:29.823087 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:20:29.823050 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:20:46.516510 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:20:46.516464 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:20:57.012012 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:20:57.011973 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:21:09.719518 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:09.719485 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:21:25.514349 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:25.514310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:21:39.032147 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:39.032046 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:21:45.776030 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.775996 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6"] Jul 16 09:21:45.779472 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.779451 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.782116 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.782096 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5f4d028a-openshift-default-dockercfg-4d59h\"" Jul 16 09:21:45.806295 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.806269 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6"] Jul 16 09:21:45.808712 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.808685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-envoy\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.808856 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.808736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-credential-socket\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.808856 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.808765 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x4lz\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-kube-api-access-2x4lz\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.808982 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.808886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-token\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.808982 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.808921 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63539f50-9e46-4fe6-8035-c4010dc51d91-istiod-ca-cert\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.808982 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.808960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-podinfo\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.809148 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.808987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-certs\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.809148 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.809017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-data\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.809148 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.809078 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-socket\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910067 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-envoy\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910208 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-credential-socket\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910208 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x4lz\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-kube-api-access-2x4lz\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910208 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-token\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910208 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63539f50-9e46-4fe6-8035-c4010dc51d91-istiod-ca-cert\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910446 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910282 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-podinfo\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910446 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-certs\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910446 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-data\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910446 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-socket\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910608 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-credential-socket\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910717 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910696 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-certs\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910820 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-socket\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.910820 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.910808 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-data\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.911056 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.911038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63539f50-9e46-4fe6-8035-c4010dc51d91-istiod-ca-cert\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.912305 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.912286 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-envoy\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.912738 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.912718 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-podinfo\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.926818 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.926791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-token\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:45.927279 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:45.927258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x4lz\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-kube-api-access-2x4lz\") pod \"e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:46.089854 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:46.089794 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:46.225087 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:46.225062 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6"] Jul 16 09:21:46.227231 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:21:46.227205 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63539f50_9e46_4fe6_8035_c4010dc51d91.slice/crio-978e21d3ea4b95b1a6c851d8ba094037962178c51c39c68626b7739c986a3585 WatchSource:0}: Error finding container 978e21d3ea4b95b1a6c851d8ba094037962178c51c39c68626b7739c986a3585: Status 404 returned error can't find the container with id 978e21d3ea4b95b1a6c851d8ba094037962178c51c39c68626b7739c986a3585 Jul 16 09:21:46.229238 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:46.229208 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:21:46.229294 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:46.229275 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:21:46.229334 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:46.229303 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:21:46.316898 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:46.316872 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" event={"ID":"63539f50-9e46-4fe6-8035-c4010dc51d91","Type":"ContainerStarted","Data":"0dc757c2beba6c7fde02722b244bf761d3cb316f7e74723986bed55076dc24db"} Jul 16 09:21:46.317014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:46.316919 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" event={"ID":"63539f50-9e46-4fe6-8035-c4010dc51d91","Type":"ContainerStarted","Data":"978e21d3ea4b95b1a6c851d8ba094037962178c51c39c68626b7739c986a3585"} Jul 16 09:21:46.337148 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:46.337096 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" podStartSLOduration=1.337078035 podStartE2EDuration="1.337078035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:21:46.336729588 +0000 UTC m=+1619.710863879" watchObservedRunningTime="2026-07-16 09:21:46.337078035 +0000 UTC m=+1619.711212326" Jul 16 09:21:47.090259 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:47.090228 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:47.094974 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:47.094952 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:47.320819 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:47.320784 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:47.321649 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:47.321632 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:49.510472 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:49.510435 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:21:52.231699 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:52.231644 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6"] Jul 16 09:21:52.232186 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:52.231916 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" podUID="63539f50-9e46-4fe6-8035-c4010dc51d91" containerName="istio-proxy" containerID="cri-o://0dc757c2beba6c7fde02722b244bf761d3cb316f7e74723986bed55076dc24db" gracePeriod=30 Jul 16 09:21:57.372010 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.371981 2574 generic.go:356] "Generic (PLEG): container finished" podID="63539f50-9e46-4fe6-8035-c4010dc51d91" containerID="0dc757c2beba6c7fde02722b244bf761d3cb316f7e74723986bed55076dc24db" exitCode=0 Jul 16 09:21:57.372395 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.372017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" event={"ID":"63539f50-9e46-4fe6-8035-c4010dc51d91","Type":"ContainerDied","Data":"0dc757c2beba6c7fde02722b244bf761d3cb316f7e74723986bed55076dc24db"} Jul 16 09:21:57.481748 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.481727 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:57.609474 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609420 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-credential-socket\") pod \"63539f50-9e46-4fe6-8035-c4010dc51d91\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " Jul 16 09:21:57.609474 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609449 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63539f50-9e46-4fe6-8035-c4010dc51d91-istiod-ca-cert\") pod \"63539f50-9e46-4fe6-8035-c4010dc51d91\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " Jul 16 09:21:57.609474 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609469 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-certs\") pod \"63539f50-9e46-4fe6-8035-c4010dc51d91\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " Jul 16 09:21:57.609690 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609488 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-token\") pod \"63539f50-9e46-4fe6-8035-c4010dc51d91\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " Jul 16 09:21:57.609690 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609508 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-podinfo\") pod \"63539f50-9e46-4fe6-8035-c4010dc51d91\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " Jul 16 09:21:57.609690 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609536 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x4lz\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-kube-api-access-2x4lz\") pod \"63539f50-9e46-4fe6-8035-c4010dc51d91\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " Jul 16 09:21:57.609690 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609586 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-socket\") pod \"63539f50-9e46-4fe6-8035-c4010dc51d91\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " Jul 16 09:21:57.609690 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609615 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-data\") pod \"63539f50-9e46-4fe6-8035-c4010dc51d91\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " Jul 16 09:21:57.609690 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609689 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-envoy\") pod \"63539f50-9e46-4fe6-8035-c4010dc51d91\" (UID: \"63539f50-9e46-4fe6-8035-c4010dc51d91\") " Jul 16 09:21:57.609988 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609709 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63539f50-9e46-4fe6-8035-c4010dc51d91" (UID: "63539f50-9e46-4fe6-8035-c4010dc51d91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:21:57.609988 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609906 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63539f50-9e46-4fe6-8035-c4010dc51d91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63539f50-9e46-4fe6-8035-c4010dc51d91" (UID: "63539f50-9e46-4fe6-8035-c4010dc51d91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 09:21:57.609988 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609917 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63539f50-9e46-4fe6-8035-c4010dc51d91" (UID: "63539f50-9e46-4fe6-8035-c4010dc51d91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:21:57.609988 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609928 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-credential-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:21:57.609988 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.609928 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63539f50-9e46-4fe6-8035-c4010dc51d91" (UID: "63539f50-9e46-4fe6-8035-c4010dc51d91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:21:57.610482 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.610456 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-data" (OuterVolumeSpecName: "istio-data") pod "63539f50-9e46-4fe6-8035-c4010dc51d91" (UID: "63539f50-9e46-4fe6-8035-c4010dc51d91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:21:57.611958 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.611927 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63539f50-9e46-4fe6-8035-c4010dc51d91" (UID: "63539f50-9e46-4fe6-8035-c4010dc51d91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 09:21:57.612039 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.611956 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-kube-api-access-2x4lz" (OuterVolumeSpecName: "kube-api-access-2x4lz") pod "63539f50-9e46-4fe6-8035-c4010dc51d91" (UID: "63539f50-9e46-4fe6-8035-c4010dc51d91"). InnerVolumeSpecName "kube-api-access-2x4lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:21:57.612039 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.611963 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-token" (OuterVolumeSpecName: "istio-token") pod "63539f50-9e46-4fe6-8035-c4010dc51d91" (UID: "63539f50-9e46-4fe6-8035-c4010dc51d91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:21:57.612039 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.611961 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63539f50-9e46-4fe6-8035-c4010dc51d91" (UID: "63539f50-9e46-4fe6-8035-c4010dc51d91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:21:57.710527 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.710499 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:21:57.710527 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.710523 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-data\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:21:57.710648 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.710532 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-envoy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:21:57.710648 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.710540 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63539f50-9e46-4fe6-8035-c4010dc51d91-istiod-ca-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:21:57.710648 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.710548 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63539f50-9e46-4fe6-8035-c4010dc51d91-workload-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:21:57.710648 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.710555 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:21:57.710648 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.710563 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63539f50-9e46-4fe6-8035-c4010dc51d91-istio-podinfo\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:21:57.710648 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:57.710570 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x4lz\" (UniqueName: \"kubernetes.io/projected/63539f50-9e46-4fe6-8035-c4010dc51d91-kube-api-access-2x4lz\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:21:58.378603 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:58.378570 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" Jul 16 09:21:58.378603 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:58.378596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6" event={"ID":"63539f50-9e46-4fe6-8035-c4010dc51d91","Type":"ContainerDied","Data":"978e21d3ea4b95b1a6c851d8ba094037962178c51c39c68626b7739c986a3585"} Jul 16 09:21:58.379169 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:58.378657 2574 scope.go:117] "RemoveContainer" containerID="0dc757c2beba6c7fde02722b244bf761d3cb316f7e74723986bed55076dc24db" Jul 16 09:21:58.403762 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:58.403738 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6"] Jul 16 09:21:58.410981 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:58.410960 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5f4d028a-openshift-default-84977f447f-5pnf6"] Jul 16 09:21:58.624631 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:58.624604 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:21:59.062546 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:21:59.062510 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63539f50-9e46-4fe6-8035-c4010dc51d91" path="/var/lib/kubelet/pods/63539f50-9e46-4fe6-8035-c4010dc51d91/volumes" Jul 16 09:22:04.680597 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.680555 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w"] Jul 16 09:22:04.681082 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.681057 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63539f50-9e46-4fe6-8035-c4010dc51d91" containerName="istio-proxy" Jul 16 09:22:04.681137 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.681086 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="63539f50-9e46-4fe6-8035-c4010dc51d91" containerName="istio-proxy" Jul 16 09:22:04.681231 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.681214 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="63539f50-9e46-4fe6-8035-c4010dc51d91" containerName="istio-proxy" Jul 16 09:22:04.686308 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.686277 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.688751 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.688726 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2d7d58ac-gw-openshift-default-dockercfg-ns44c\"" Jul 16 09:22:04.698224 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.698199 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w"] Jul 16 09:22:04.762406 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.762384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-envoy\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.762516 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.762416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmj4z\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-kube-api-access-zmj4z\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.762516 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.762442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-podinfo\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.762516 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.762464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-credential-socket\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.762516 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.762487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-token\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.762516 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.762508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-data\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.762693 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.762524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237bcefb-feeb-48db-ba9c-ad9aa8274545-istiod-ca-cert\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.762693 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.762608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-socket\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.762693 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.762640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-certs\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.863868 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.863847 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-socket\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.863990 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.863876 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-certs\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.863990 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.863912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-envoy\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.863990 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.863937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmj4z\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-kube-api-access-zmj4z\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.863990 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.863971 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-podinfo\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.864244 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.863998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-credential-socket\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.864244 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.864032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-token\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.864244 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.864061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-data\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.864244 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.864084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237bcefb-feeb-48db-ba9c-ad9aa8274545-istiod-ca-cert\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.864493 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.864268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-socket\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.864493 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.864308 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-certs\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.864493 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.864426 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-credential-socket\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.864648 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.864501 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-data\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.864703 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.864678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237bcefb-feeb-48db-ba9c-ad9aa8274545-istiod-ca-cert\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.866390 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.866366 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-envoy\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.866472 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.866429 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-podinfo\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.873128 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.873110 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-token\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.873240 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.873225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmj4z\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-kube-api-access-zmj4z\") pod \"e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:04.998800 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:04.998748 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:05.132544 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:05.132519 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w"] Jul 16 09:22:05.135044 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:22:05.135015 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237bcefb_feeb_48db_ba9c_ad9aa8274545.slice/crio-cd63698263d063972267160796fa9004ee4a122e3cb9d5983d3bc714417d5674 WatchSource:0}: Error finding container cd63698263d063972267160796fa9004ee4a122e3cb9d5983d3bc714417d5674: Status 404 returned error can't find the container with id cd63698263d063972267160796fa9004ee4a122e3cb9d5983d3bc714417d5674 Jul 16 09:22:05.137559 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:05.137521 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:22:05.137654 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:05.137599 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:22:05.137654 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:05.137642 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:22:05.409170 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:05.409124 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" event={"ID":"237bcefb-feeb-48db-ba9c-ad9aa8274545","Type":"ContainerStarted","Data":"08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07"} Jul 16 09:22:05.409357 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:05.409203 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" event={"ID":"237bcefb-feeb-48db-ba9c-ad9aa8274545","Type":"ContainerStarted","Data":"cd63698263d063972267160796fa9004ee4a122e3cb9d5983d3bc714417d5674"} Jul 16 09:22:05.435049 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:05.435000 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" podStartSLOduration=1.434981715 podStartE2EDuration="1.434981715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:22:05.434241666 +0000 UTC m=+1638.808375957" watchObservedRunningTime="2026-07-16 09:22:05.434981715 +0000 UTC m=+1638.809116001" Jul 16 09:22:05.999591 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:05.999562 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:07.003594 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:07.003564 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:07.418908 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:07.418868 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:07.419973 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:07.419948 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:08.629239 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:08.629206 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:22:10.653731 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:10.653685 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w"] Jul 16 09:22:10.654273 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:10.653993 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" podUID="237bcefb-feeb-48db-ba9c-ad9aa8274545" containerName="istio-proxy" containerID="cri-o://08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07" gracePeriod=30 Jul 16 09:22:16.801417 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.801395 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:16.873837 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.873814 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-token\") pod \"237bcefb-feeb-48db-ba9c-ad9aa8274545\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " Jul 16 09:22:16.873929 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.873863 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-envoy\") pod \"237bcefb-feeb-48db-ba9c-ad9aa8274545\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " Jul 16 09:22:16.873929 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.873892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-podinfo\") pod \"237bcefb-feeb-48db-ba9c-ad9aa8274545\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " Jul 16 09:22:16.873929 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.873920 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-credential-socket\") pod \"237bcefb-feeb-48db-ba9c-ad9aa8274545\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " Jul 16 09:22:16.874075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.873939 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-data\") pod \"237bcefb-feeb-48db-ba9c-ad9aa8274545\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " Jul 16 09:22:16.874075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.873990 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmj4z\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-kube-api-access-zmj4z\") pod \"237bcefb-feeb-48db-ba9c-ad9aa8274545\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " Jul 16 09:22:16.874075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.874015 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237bcefb-feeb-48db-ba9c-ad9aa8274545-istiod-ca-cert\") pod \"237bcefb-feeb-48db-ba9c-ad9aa8274545\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " Jul 16 09:22:16.874075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.874038 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-certs\") pod \"237bcefb-feeb-48db-ba9c-ad9aa8274545\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " Jul 16 09:22:16.874075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.874068 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-socket\") pod \"237bcefb-feeb-48db-ba9c-ad9aa8274545\" (UID: \"237bcefb-feeb-48db-ba9c-ad9aa8274545\") " Jul 16 09:22:16.874386 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.874280 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "237bcefb-feeb-48db-ba9c-ad9aa8274545" (UID: "237bcefb-feeb-48db-ba9c-ad9aa8274545"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:16.874545 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.874481 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "237bcefb-feeb-48db-ba9c-ad9aa8274545" (UID: "237bcefb-feeb-48db-ba9c-ad9aa8274545"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:16.874545 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.874497 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "237bcefb-feeb-48db-ba9c-ad9aa8274545" (UID: "237bcefb-feeb-48db-ba9c-ad9aa8274545"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:16.874653 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.874604 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237bcefb-feeb-48db-ba9c-ad9aa8274545-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "237bcefb-feeb-48db-ba9c-ad9aa8274545" (UID: "237bcefb-feeb-48db-ba9c-ad9aa8274545"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 09:22:16.874875 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.874845 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-data" (OuterVolumeSpecName: "istio-data") pod "237bcefb-feeb-48db-ba9c-ad9aa8274545" (UID: "237bcefb-feeb-48db-ba9c-ad9aa8274545"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:16.876118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.876087 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "237bcefb-feeb-48db-ba9c-ad9aa8274545" (UID: "237bcefb-feeb-48db-ba9c-ad9aa8274545"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:16.876323 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.876304 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-kube-api-access-zmj4z" (OuterVolumeSpecName: "kube-api-access-zmj4z") pod "237bcefb-feeb-48db-ba9c-ad9aa8274545" (UID: "237bcefb-feeb-48db-ba9c-ad9aa8274545"). InnerVolumeSpecName "kube-api-access-zmj4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:22:16.876434 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.876415 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "237bcefb-feeb-48db-ba9c-ad9aa8274545" (UID: "237bcefb-feeb-48db-ba9c-ad9aa8274545"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 09:22:16.876498 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.876456 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-token" (OuterVolumeSpecName: "istio-token") pod "237bcefb-feeb-48db-ba9c-ad9aa8274545" (UID: "237bcefb-feeb-48db-ba9c-ad9aa8274545"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:22:16.974721 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.974699 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-podinfo\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:16.974721 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.974718 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-credential-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:16.974843 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.974727 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-data\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:16.974843 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.974736 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmj4z\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-kube-api-access-zmj4z\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:16.974843 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.974744 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237bcefb-feeb-48db-ba9c-ad9aa8274545-istiod-ca-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:16.974843 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.974753 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:16.974843 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.974760 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-workload-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:16.974843 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.974768 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:16.974843 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:16.974775 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237bcefb-feeb-48db-ba9c-ad9aa8274545-istio-envoy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:17.468503 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:17.468464 2574 generic.go:356] "Generic (PLEG): container finished" podID="237bcefb-feeb-48db-ba9c-ad9aa8274545" containerID="08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07" exitCode=0 Jul 16 09:22:17.468650 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:17.468522 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" Jul 16 09:22:17.468650 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:17.468548 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" event={"ID":"237bcefb-feeb-48db-ba9c-ad9aa8274545","Type":"ContainerDied","Data":"08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07"} Jul 16 09:22:17.468650 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:17.468607 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w" event={"ID":"237bcefb-feeb-48db-ba9c-ad9aa8274545","Type":"ContainerDied","Data":"cd63698263d063972267160796fa9004ee4a122e3cb9d5983d3bc714417d5674"} Jul 16 09:22:17.468650 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:17.468638 2574 scope.go:117] "RemoveContainer" containerID="08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07" Jul 16 09:22:17.478676 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:17.478659 2574 scope.go:117] "RemoveContainer" containerID="08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07" Jul 16 09:22:17.478942 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:22:17.478924 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07\": container with ID starting with 08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07 not found: ID does not exist" containerID="08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07" Jul 16 09:22:17.478991 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:17.478950 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07"} err="failed to get container status \"08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07\": rpc error: code = NotFound desc = could not find container \"08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07\": container with ID starting with 08b57a96faa31e91674d52ace302602af859b350f943eaaa09b0c2df40afcc07 not found: ID does not exist" Jul 16 09:22:17.489106 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:17.489078 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w"] Jul 16 09:22:17.493584 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:17.493565 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2d7d58ac-gw-openshift-default-676d86cc59-wxb4w"] Jul 16 09:22:19.021680 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:19.021644 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:22:19.055023 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:19.054998 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="237bcefb-feeb-48db-ba9c-ad9aa8274545" path="/var/lib/kubelet/pods/237bcefb-feeb-48db-ba9c-ad9aa8274545/volumes" Jul 16 09:22:23.614680 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.614642 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk"] Jul 16 09:22:23.615172 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.615128 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="237bcefb-feeb-48db-ba9c-ad9aa8274545" containerName="istio-proxy" Jul 16 09:22:23.615172 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.615171 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="237bcefb-feeb-48db-ba9c-ad9aa8274545" containerName="istio-proxy" Jul 16 09:22:23.615291 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.615281 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="237bcefb-feeb-48db-ba9c-ad9aa8274545" containerName="istio-proxy" Jul 16 09:22:23.620243 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.620214 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.623042 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.623021 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ed97d282-openshift-default-dockercfg-ltn5v\"" Jul 16 09:22:23.631195 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.631167 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk"] Jul 16 09:22:23.724969 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.724940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb5151-fa65-4ee5-b919-f39298b78f07-istiod-ca-cert\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.725084 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.725000 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-podinfo\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.725084 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.725029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqtb4\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-kube-api-access-kqtb4\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.725084 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.725057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-data\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.725084 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.725079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-token\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.725322 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.725174 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-envoy\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.725322 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.725231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-socket\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.725322 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.725259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-credential-socket\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.725322 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.725282 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-certs\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.825912 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.825891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb5151-fa65-4ee5-b919-f39298b78f07-istiod-ca-cert\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.825931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-podinfo\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.825947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqtb4\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-kube-api-access-kqtb4\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.825964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-data\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.825987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-token\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826262 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.826124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-envoy\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826262 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.826201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-socket\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826262 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.826226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-credential-socket\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826262 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.826250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-certs\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826459 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.826418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-data\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826604 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.826581 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-socket\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826665 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.826602 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-credential-socket\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826724 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.826672 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-certs\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.826776 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.826723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb5151-fa65-4ee5-b919-f39298b78f07-istiod-ca-cert\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.828730 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.828704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-envoy\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.828886 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.828870 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-podinfo\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.840235 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.840210 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqtb4\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-kube-api-access-kqtb4\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.840637 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.840603 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-token\") pod \"e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:23.932252 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:23.932179 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:24.070413 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:24.070389 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk"] Jul 16 09:22:24.072945 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:22:24.072903 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aeb5151_fa65_4ee5_b919_f39298b78f07.slice/crio-9e71b61d29bab28c47ea6af61a7ed050a3cbd7da608d26aae7d13d3dc08cf6ba WatchSource:0}: Error finding container 9e71b61d29bab28c47ea6af61a7ed050a3cbd7da608d26aae7d13d3dc08cf6ba: Status 404 returned error can't find the container with id 9e71b61d29bab28c47ea6af61a7ed050a3cbd7da608d26aae7d13d3dc08cf6ba Jul 16 09:22:24.074943 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:24.074911 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:22:24.075024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:24.075003 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:22:24.075065 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:24.075031 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:22:24.502202 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:24.502149 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" event={"ID":"4aeb5151-fa65-4ee5-b919-f39298b78f07","Type":"ContainerStarted","Data":"b6d983a522f26c5121e148770b6d63c61c86a828266c9471e07616eabf51c04e"} Jul 16 09:22:24.502360 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:24.502224 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" event={"ID":"4aeb5151-fa65-4ee5-b919-f39298b78f07","Type":"ContainerStarted","Data":"9e71b61d29bab28c47ea6af61a7ed050a3cbd7da608d26aae7d13d3dc08cf6ba"} Jul 16 09:22:24.522666 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:24.522613 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" podStartSLOduration=1.5225929329999999 podStartE2EDuration="1.522592933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:22:24.520550967 +0000 UTC m=+1657.894685259" watchObservedRunningTime="2026-07-16 09:22:24.522592933 +0000 UTC m=+1657.896727228" Jul 16 09:22:24.932351 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:24.932298 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:25.938002 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:25.937973 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:26.511115 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:26.511083 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:26.512046 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:26.512026 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:27.916693 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:27.916653 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:22:29.323939 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:29.323895 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk"] Jul 16 09:22:29.526419 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:29.526375 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" podUID="4aeb5151-fa65-4ee5-b919-f39298b78f07" containerName="istio-proxy" containerID="cri-o://b6d983a522f26c5121e148770b6d63c61c86a828266c9471e07616eabf51c04e" gracePeriod=30 Jul 16 09:22:34.713364 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:34.713334 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:22:35.558861 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.558830 2574 generic.go:356] "Generic (PLEG): container finished" podID="4aeb5151-fa65-4ee5-b919-f39298b78f07" containerID="b6d983a522f26c5121e148770b6d63c61c86a828266c9471e07616eabf51c04e" exitCode=0 Jul 16 09:22:35.559016 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.558908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" event={"ID":"4aeb5151-fa65-4ee5-b919-f39298b78f07","Type":"ContainerDied","Data":"b6d983a522f26c5121e148770b6d63c61c86a828266c9471e07616eabf51c04e"} Jul 16 09:22:35.586299 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.586273 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:35.629340 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629319 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-certs\") pod \"4aeb5151-fa65-4ee5-b919-f39298b78f07\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " Jul 16 09:22:35.629484 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629361 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-credential-socket\") pod \"4aeb5151-fa65-4ee5-b919-f39298b78f07\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " Jul 16 09:22:35.629484 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629394 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-podinfo\") pod \"4aeb5151-fa65-4ee5-b919-f39298b78f07\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " Jul 16 09:22:35.629484 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629419 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqtb4\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-kube-api-access-kqtb4\") pod \"4aeb5151-fa65-4ee5-b919-f39298b78f07\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " Jul 16 09:22:35.629484 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629476 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-token\") pod \"4aeb5151-fa65-4ee5-b919-f39298b78f07\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " Jul 16 09:22:35.629687 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629611 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-socket\") pod \"4aeb5151-fa65-4ee5-b919-f39298b78f07\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " Jul 16 09:22:35.629687 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629623 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4aeb5151-fa65-4ee5-b919-f39298b78f07" (UID: "4aeb5151-fa65-4ee5-b919-f39298b78f07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:35.629687 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629661 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-envoy\") pod \"4aeb5151-fa65-4ee5-b919-f39298b78f07\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " Jul 16 09:22:35.629832 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629684 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4aeb5151-fa65-4ee5-b919-f39298b78f07" (UID: "4aeb5151-fa65-4ee5-b919-f39298b78f07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:35.629832 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629692 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-data\") pod \"4aeb5151-fa65-4ee5-b919-f39298b78f07\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " Jul 16 09:22:35.629832 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629774 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb5151-fa65-4ee5-b919-f39298b78f07-istiod-ca-cert\") pod \"4aeb5151-fa65-4ee5-b919-f39298b78f07\" (UID: \"4aeb5151-fa65-4ee5-b919-f39298b78f07\") " Jul 16 09:22:35.629986 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.629835 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4aeb5151-fa65-4ee5-b919-f39298b78f07" (UID: "4aeb5151-fa65-4ee5-b919-f39298b78f07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:35.630076 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.630055 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:35.630142 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.630084 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-credential-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:35.630142 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.630099 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-workload-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:35.630288 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.630209 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aeb5151-fa65-4ee5-b919-f39298b78f07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4aeb5151-fa65-4ee5-b919-f39298b78f07" (UID: "4aeb5151-fa65-4ee5-b919-f39298b78f07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 09:22:35.630515 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.630490 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-data" (OuterVolumeSpecName: "istio-data") pod "4aeb5151-fa65-4ee5-b919-f39298b78f07" (UID: "4aeb5151-fa65-4ee5-b919-f39298b78f07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:35.632027 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.631998 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4aeb5151-fa65-4ee5-b919-f39298b78f07" (UID: "4aeb5151-fa65-4ee5-b919-f39298b78f07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 09:22:35.632343 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.632322 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-token" (OuterVolumeSpecName: "istio-token") pod "4aeb5151-fa65-4ee5-b919-f39298b78f07" (UID: "4aeb5151-fa65-4ee5-b919-f39298b78f07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:22:35.632584 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.632562 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-kube-api-access-kqtb4" (OuterVolumeSpecName: "kube-api-access-kqtb4") pod "4aeb5151-fa65-4ee5-b919-f39298b78f07" (UID: "4aeb5151-fa65-4ee5-b919-f39298b78f07"). InnerVolumeSpecName "kube-api-access-kqtb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:22:35.632666 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.632643 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4aeb5151-fa65-4ee5-b919-f39298b78f07" (UID: "4aeb5151-fa65-4ee5-b919-f39298b78f07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:22:35.730599 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.730578 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:35.730599 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.730598 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-envoy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:35.731005 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.730606 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-data\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:35.731005 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.730614 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aeb5151-fa65-4ee5-b919-f39298b78f07-istiod-ca-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:35.731005 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.730622 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aeb5151-fa65-4ee5-b919-f39298b78f07-istio-podinfo\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:35.731005 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:35.730629 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqtb4\" (UniqueName: \"kubernetes.io/projected/4aeb5151-fa65-4ee5-b919-f39298b78f07-kube-api-access-kqtb4\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:22:36.565486 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:36.565454 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" Jul 16 09:22:36.565486 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:36.565475 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk" event={"ID":"4aeb5151-fa65-4ee5-b919-f39298b78f07","Type":"ContainerDied","Data":"9e71b61d29bab28c47ea6af61a7ed050a3cbd7da608d26aae7d13d3dc08cf6ba"} Jul 16 09:22:36.565694 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:36.565553 2574 scope.go:117] "RemoveContainer" containerID="b6d983a522f26c5121e148770b6d63c61c86a828266c9471e07616eabf51c04e" Jul 16 09:22:36.588474 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:36.588448 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk"] Jul 16 09:22:36.594509 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:36.594484 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ed97d282-openshift-default-5fffff79ff-x6zvk"] Jul 16 09:22:37.055868 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:22:37.055842 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4aeb5151-fa65-4ee5-b919-f39298b78f07" path="/var/lib/kubelet/pods/4aeb5151-fa65-4ee5-b919-f39298b78f07/volumes" Jul 16 09:24:06.515590 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:06.515555 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:24:13.930654 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:13.930607 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:24:21.739202 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:21.739163 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:24:22.513938 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.513909 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd"] Jul 16 09:24:22.514254 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.514239 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4aeb5151-fa65-4ee5-b919-f39298b78f07" containerName="istio-proxy" Jul 16 09:24:22.514321 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.514255 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4aeb5151-fa65-4ee5-b919-f39298b78f07" containerName="istio-proxy" Jul 16 09:24:22.514321 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.514305 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4aeb5151-fa65-4ee5-b919-f39298b78f07" containerName="istio-proxy" Jul 16 09:24:22.517204 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.517187 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.521254 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.521235 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8e4c8858-openshift-default-dockercfg-jvwh7\"" Jul 16 09:24:22.543203 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.543179 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd"] Jul 16 09:24:22.651330 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.651303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcc2d\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-kube-api-access-jcc2d\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.651440 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.651337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-data\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.651440 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.651355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-certs\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.651440 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.651374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-socket\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.651544 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.651441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-credential-socket\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.651544 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.651480 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-envoy\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.651544 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.651499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-istio-token\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.651544 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.651527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97475e52-cff5-483b-a809-d1e5776caa34-istiod-ca-cert\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.651668 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.651560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97475e52-cff5-483b-a809-d1e5776caa34-istio-podinfo\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.752733 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.752702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-istio-token\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.752742 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97475e52-cff5-483b-a809-d1e5776caa34-istiod-ca-cert\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.752783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97475e52-cff5-483b-a809-d1e5776caa34-istio-podinfo\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.752846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcc2d\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-kube-api-access-jcc2d\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.752891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-data\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.752919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-certs\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.752953 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-socket\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.753069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-credential-socket\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.753104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-envoy\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753536 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.753431 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-certs\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753536 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.753463 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-data\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753605 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.753561 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-socket\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753652 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.753625 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-credential-socket\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.753688 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.753676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97475e52-cff5-483b-a809-d1e5776caa34-istiod-ca-cert\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.755323 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.755302 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-envoy\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.755566 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.755547 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97475e52-cff5-483b-a809-d1e5776caa34-istio-podinfo\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.762525 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.762496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-istio-token\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.762758 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.762739 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcc2d\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-kube-api-access-jcc2d\") pod \"e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.826485 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.826434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:22.958750 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.958726 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd"] Jul 16 09:24:22.961451 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:24:22.961420 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97475e52_cff5_483b_a809_d1e5776caa34.slice/crio-a7e39f97037a69e81c4cb911816786ebba4151fd65b01cdc5e96bf17a53a0630 WatchSource:0}: Error finding container a7e39f97037a69e81c4cb911816786ebba4151fd65b01cdc5e96bf17a53a0630: Status 404 returned error can't find the container with id a7e39f97037a69e81c4cb911816786ebba4151fd65b01cdc5e96bf17a53a0630 Jul 16 09:24:22.963680 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.963647 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:24:22.963749 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.963710 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:24:22.963749 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:22.963737 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:24:23.038869 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:23.038808 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" event={"ID":"97475e52-cff5-483b-a809-d1e5776caa34","Type":"ContainerStarted","Data":"a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf"} Jul 16 09:24:23.038869 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:23.038862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" event={"ID":"97475e52-cff5-483b-a809-d1e5776caa34","Type":"ContainerStarted","Data":"a7e39f97037a69e81c4cb911816786ebba4151fd65b01cdc5e96bf17a53a0630"} Jul 16 09:24:23.066730 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:23.066685 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" podStartSLOduration=1.066669644 podStartE2EDuration="1.066669644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:24:23.064828709 +0000 UTC m=+1776.438963000" watchObservedRunningTime="2026-07-16 09:24:23.066669644 +0000 UTC m=+1776.440803934" Jul 16 09:24:23.826695 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:23.826668 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:24.831446 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:24.831418 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:25.048202 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:25.048150 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:25.048981 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:25.048963 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:31.431777 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:31.431741 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:24:40.049744 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:40.049700 2574 patch_prober.go:28] interesting pod/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 16 09:24:40.050241 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:40.049777 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 16 09:24:40.117782 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:40.117743 2574 generic.go:356] "Generic (PLEG): container finished" podID="97475e52-cff5-483b-a809-d1e5776caa34" containerID="a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf" exitCode=0 Jul 16 09:24:40.117948 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:40.117818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" event={"ID":"97475e52-cff5-483b-a809-d1e5776caa34","Type":"ContainerDied","Data":"a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf"} Jul 16 09:24:40.118280 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:40.118265 2574 scope.go:117] "RemoveContainer" containerID="a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf" Jul 16 09:24:40.119340 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:40.119310 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:24:40.119464 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:40.119380 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:24:40.119464 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:40.119412 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:24:41.126852 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:41.126798 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" event={"ID":"97475e52-cff5-483b-a809-d1e5776caa34","Type":"ContainerStarted","Data":"b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade"} Jul 16 09:24:41.827200 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:41.827145 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:41.827615 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:41.827527 2574 patch_prober.go:28] interesting pod/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 16 09:24:41.827783 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:41.827760 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 16 09:24:42.827884 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:42.827841 2574 patch_prober.go:28] interesting pod/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 16 09:24:42.828550 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:42.828514 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 16 09:24:43.827727 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:43.827256 2574 patch_prober.go:28] interesting pod/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 16 09:24:43.827727 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:43.827332 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 16 09:24:44.827680 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:44.827634 2574 patch_prober.go:28] interesting pod/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 16 09:24:44.827900 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:44.827705 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 16 09:24:45.831371 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:45.831337 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:45.831833 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:45.831622 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:45.832433 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:45.832416 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:46.812233 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:46.812197 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:24:47.114824 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:47.114796 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:24:47.117959 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:47.117941 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:24:50.231442 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:50.231404 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd"] Jul 16 09:24:50.231897 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:50.231686 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" containerID="cri-o://b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade" gracePeriod=30 Jul 16 09:24:55.471967 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.471940 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:55.541725 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541666 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcc2d\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-kube-api-access-jcc2d\") pod \"97475e52-cff5-483b-a809-d1e5776caa34\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " Jul 16 09:24:55.541725 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541710 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-socket\") pod \"97475e52-cff5-483b-a809-d1e5776caa34\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " Jul 16 09:24:55.541877 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541735 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97475e52-cff5-483b-a809-d1e5776caa34-istio-podinfo\") pod \"97475e52-cff5-483b-a809-d1e5776caa34\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " Jul 16 09:24:55.541877 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541763 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-certs\") pod \"97475e52-cff5-483b-a809-d1e5776caa34\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " Jul 16 09:24:55.541877 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-istio-token\") pod \"97475e52-cff5-483b-a809-d1e5776caa34\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " Jul 16 09:24:55.541877 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541849 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-envoy\") pod \"97475e52-cff5-483b-a809-d1e5776caa34\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " Jul 16 09:24:55.542075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541904 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-data\") pod \"97475e52-cff5-483b-a809-d1e5776caa34\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " Jul 16 09:24:55.542075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541937 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-credential-socket\") pod \"97475e52-cff5-483b-a809-d1e5776caa34\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " Jul 16 09:24:55.542075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541968 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97475e52-cff5-483b-a809-d1e5776caa34-istiod-ca-cert\") pod \"97475e52-cff5-483b-a809-d1e5776caa34\" (UID: \"97475e52-cff5-483b-a809-d1e5776caa34\") " Jul 16 09:24:55.542075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.541987 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97475e52-cff5-483b-a809-d1e5776caa34" (UID: "97475e52-cff5-483b-a809-d1e5776caa34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:24:55.542305 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.542094 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97475e52-cff5-483b-a809-d1e5776caa34" (UID: "97475e52-cff5-483b-a809-d1e5776caa34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:24:55.542382 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.542348 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:24:55.542382 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.542374 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-workload-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:24:55.542515 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.542488 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97475e52-cff5-483b-a809-d1e5776caa34-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97475e52-cff5-483b-a809-d1e5776caa34" (UID: "97475e52-cff5-483b-a809-d1e5776caa34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 09:24:55.542638 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.542541 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97475e52-cff5-483b-a809-d1e5776caa34" (UID: "97475e52-cff5-483b-a809-d1e5776caa34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:24:55.543129 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.543093 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-data" (OuterVolumeSpecName: "istio-data") pod "97475e52-cff5-483b-a809-d1e5776caa34" (UID: "97475e52-cff5-483b-a809-d1e5776caa34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:24:55.543992 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.543957 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97475e52-cff5-483b-a809-d1e5776caa34" (UID: "97475e52-cff5-483b-a809-d1e5776caa34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:24:55.544075 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.543995 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97475e52-cff5-483b-a809-d1e5776caa34-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97475e52-cff5-483b-a809-d1e5776caa34" (UID: "97475e52-cff5-483b-a809-d1e5776caa34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 09:24:55.544115 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.544093 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-istio-token" (OuterVolumeSpecName: "istio-token") pod "97475e52-cff5-483b-a809-d1e5776caa34" (UID: "97475e52-cff5-483b-a809-d1e5776caa34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:24:55.544115 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.544102 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-kube-api-access-jcc2d" (OuterVolumeSpecName: "kube-api-access-jcc2d") pod "97475e52-cff5-483b-a809-d1e5776caa34" (UID: "97475e52-cff5-483b-a809-d1e5776caa34"). InnerVolumeSpecName "kube-api-access-jcc2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:24:55.643668 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.643644 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcc2d\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-kube-api-access-jcc2d\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:24:55.643668 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.643665 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97475e52-cff5-483b-a809-d1e5776caa34-istio-podinfo\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:24:55.643798 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.643675 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97475e52-cff5-483b-a809-d1e5776caa34-istio-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:24:55.643798 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.643683 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-envoy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:24:55.643798 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.643690 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-istio-data\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:24:55.643798 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.643698 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97475e52-cff5-483b-a809-d1e5776caa34-credential-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:24:55.643798 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:55.643708 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97475e52-cff5-483b-a809-d1e5776caa34-istiod-ca-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:24:56.201006 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.200978 2574 generic.go:356] "Generic (PLEG): container finished" podID="97475e52-cff5-483b-a809-d1e5776caa34" containerID="b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade" exitCode=0 Jul 16 09:24:56.201132 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.201026 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" Jul 16 09:24:56.201132 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.201061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" event={"ID":"97475e52-cff5-483b-a809-d1e5776caa34","Type":"ContainerDied","Data":"b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade"} Jul 16 09:24:56.201132 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.201120 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd" event={"ID":"97475e52-cff5-483b-a809-d1e5776caa34","Type":"ContainerDied","Data":"a7e39f97037a69e81c4cb911816786ebba4151fd65b01cdc5e96bf17a53a0630"} Jul 16 09:24:56.201274 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.201173 2574 scope.go:117] "RemoveContainer" containerID="b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade" Jul 16 09:24:56.210622 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.210604 2574 scope.go:117] "RemoveContainer" containerID="a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf" Jul 16 09:24:56.218322 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.218301 2574 scope.go:117] "RemoveContainer" containerID="b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade" Jul 16 09:24:56.218562 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:24:56.218542 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade\": container with ID starting with b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade not found: ID does not exist" containerID="b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade" Jul 16 09:24:56.218607 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.218569 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade"} err="failed to get container status \"b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade\": rpc error: code = NotFound desc = could not find container \"b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade\": container with ID starting with b26ed83499cd570b1df1e5482631c01c94ea47f3a6f3936cdd78344a89a1dade not found: ID does not exist" Jul 16 09:24:56.218607 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.218586 2574 scope.go:117] "RemoveContainer" containerID="a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf" Jul 16 09:24:56.218854 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:24:56.218835 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf\": container with ID starting with a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf not found: ID does not exist" containerID="a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf" Jul 16 09:24:56.218909 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.218859 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf"} err="failed to get container status \"a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf\": rpc error: code = NotFound desc = could not find container \"a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf\": container with ID starting with a8bb6591b3a4fe64059dc4bfc789031320e5990ac843fe0bbaeb26b21ee144bf not found: ID does not exist" Jul 16 09:24:56.222460 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.222437 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd"] Jul 16 09:24:56.228092 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:56.228072 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8e4c8858-openshift-default-85d8b74979-wfmxd"] Jul 16 09:24:57.055546 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:24:57.055503 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97475e52-cff5-483b-a809-d1e5776caa34" path="/var/lib/kubelet/pods/97475e52-cff5-483b-a809-d1e5776caa34/volumes" Jul 16 09:25:15.915777 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:25:15.915745 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:25:29.108809 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:25:29.108771 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:26:05.819350 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.819300 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb"] Jul 16 09:26:05.819880 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.819629 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" Jul 16 09:26:05.819880 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.819640 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" Jul 16 09:26:05.819880 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.819663 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" Jul 16 09:26:05.819880 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.819670 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" Jul 16 09:26:05.819880 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.819717 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" Jul 16 09:26:05.822375 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.822352 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.824996 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.824972 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-00d870-openshift-default-dockercfg-rmc6f\"" Jul 16 09:26:05.833910 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.833886 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb"] Jul 16 09:26:05.872641 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.872616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-envoy\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.872733 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.872645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-podinfo\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.872733 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.872665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26cwp\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-kube-api-access-26cwp\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.872733 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.872707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-certs\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.872836 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.872747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-socket\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.872836 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.872789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-data\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.872836 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.872811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istiod-ca-cert\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.872927 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.872868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-token\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.872927 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.872912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-credential-socket\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974254 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-token\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974388 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-credential-socket\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974388 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-envoy\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974388 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-podinfo\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974388 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26cwp\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-kube-api-access-26cwp\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974388 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-certs\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974666 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-socket\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974666 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-data\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974666 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istiod-ca-cert\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974794 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974662 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-credential-socket\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974794 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974741 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-certs\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974858 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974837 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-socket\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.974932 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.974903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-data\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.975271 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.975255 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istiod-ca-cert\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.976722 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.976700 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-envoy\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.976821 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.976805 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-podinfo\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.982302 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.982277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-token\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:05.982665 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:05.982648 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26cwp\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-kube-api-access-26cwp\") pod \"e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:06.134964 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:06.134934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:06.274460 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:06.274428 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb"] Jul 16 09:26:06.277635 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:26:06.277592 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f59953_ffd8_40b9_85ce_693fdbb3fb4a.slice/crio-ca1ae8083059e8a1132efa43bb1632bdc61dc7404808acf0e390c04b660b6fbd WatchSource:0}: Error finding container ca1ae8083059e8a1132efa43bb1632bdc61dc7404808acf0e390c04b660b6fbd: Status 404 returned error can't find the container with id ca1ae8083059e8a1132efa43bb1632bdc61dc7404808acf0e390c04b660b6fbd Jul 16 09:26:06.279909 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:06.279869 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:26:06.279975 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:06.279956 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:26:06.280014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:06.279996 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:26:06.512265 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:06.512181 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" event={"ID":"98f59953-ffd8-40b9-85ce-693fdbb3fb4a","Type":"ContainerStarted","Data":"81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c"} Jul 16 09:26:06.512265 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:06.512241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" event={"ID":"98f59953-ffd8-40b9-85ce-693fdbb3fb4a","Type":"ContainerStarted","Data":"ca1ae8083059e8a1132efa43bb1632bdc61dc7404808acf0e390c04b660b6fbd"} Jul 16 09:26:06.541456 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:06.541399 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" podStartSLOduration=1.541378311 podStartE2EDuration="1.541378311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:26:06.539242407 +0000 UTC m=+1879.913376696" watchObservedRunningTime="2026-07-16 09:26:06.541378311 +0000 UTC m=+1879.915512595" Jul 16 09:26:07.135988 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:07.135896 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:07.140984 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:07.140956 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:07.517221 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:07.517132 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:07.517995 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:07.517971 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:16.005641 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:16.005588 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:26:23.448417 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:23.447441 2574 patch_prober.go:28] interesting pod/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": read tcp 10.133.0.2:50700->10.133.0.29:15021: read: connection reset by peer" start-of-body= Jul 16 09:26:23.448417 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:23.447521 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": read tcp 10.133.0.2:50700->10.133.0.29:15021: read: connection reset by peer" Jul 16 09:26:23.589958 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:23.589928 2574 generic.go:356] "Generic (PLEG): container finished" podID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerID="81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c" exitCode=0 Jul 16 09:26:23.590067 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:23.590003 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" event={"ID":"98f59953-ffd8-40b9-85ce-693fdbb3fb4a","Type":"ContainerDied","Data":"81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c"} Jul 16 09:26:23.590412 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:23.590398 2574 scope.go:117] "RemoveContainer" containerID="81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c" Jul 16 09:26:23.591371 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:23.591345 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:26:23.591427 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:23.591413 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:26:23.591463 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:23.591445 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:26:24.596853 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:24.596821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" event={"ID":"98f59953-ffd8-40b9-85ce-693fdbb3fb4a","Type":"ContainerStarted","Data":"6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987"} Jul 16 09:26:25.135304 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:25.135272 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:25.135603 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:25.135574 2574 patch_prober.go:28] interesting pod/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 16 09:26:25.135687 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:25.135633 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 16 09:26:26.136462 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:26.136005 2574 patch_prober.go:28] interesting pod/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 16 09:26:26.136462 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:26.136070 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 16 09:26:26.921794 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:26.921747 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:26:27.136637 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:27.136237 2574 patch_prober.go:28] interesting pod/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 16 09:26:27.136637 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:27.136292 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 16 09:26:28.135514 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:28.135476 2574 patch_prober.go:28] interesting pod/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 16 09:26:28.135800 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:28.135770 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 16 09:26:29.139605 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:29.139568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:29.140166 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:29.140131 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:29.140951 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:29.140927 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:26:55.275317 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.275275 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf"] Jul 16 09:26:55.275842 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.275825 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="97475e52-cff5-483b-a809-d1e5776caa34" containerName="istio-proxy" Jul 16 09:26:55.279255 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.279233 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.282618 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.282593 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-00d870\"/\"default-dockercfg-cmqf5\"" Jul 16 09:26:55.282770 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.282635 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-00d870\"/\"openshift-service-ca.crt\"" Jul 16 09:26:55.282770 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.282723 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-00d870\"/\"kube-root-ca.crt\"" Jul 16 09:26:55.282770 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.282725 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-00d870\"/\"test-model-00d870-kserve-self-signed-certs\"" Jul 16 09:26:55.292292 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.292266 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf"] Jul 16 09:26:55.373909 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.373888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-tmp-dir\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.374007 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.373916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjcz\" (UniqueName: \"kubernetes.io/projected/d0e7b33d-6524-410e-90bc-25212001726b-kube-api-access-xzjcz\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.374007 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.373988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-model-cache\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.374092 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.374006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e7b33d-6524-410e-90bc-25212001726b-tls-certs\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.374132 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.374074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-dshm\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.374183 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.374142 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-home\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.474850 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.474825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-home\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.474850 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.474857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-tmp-dir\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.475024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.474874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjcz\" (UniqueName: \"kubernetes.io/projected/d0e7b33d-6524-410e-90bc-25212001726b-kube-api-access-xzjcz\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.475024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.474919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-model-cache\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.475024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.474934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e7b33d-6524-410e-90bc-25212001726b-tls-certs\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.475024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.474968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-dshm\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.475289 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.475268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-tmp-dir\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.475334 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.475314 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-model-cache\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.475380 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.475332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-home\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.477104 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.477077 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-dshm\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.477364 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.477347 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e7b33d-6524-410e-90bc-25212001726b-tls-certs\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.484978 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.484956 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjcz\" (UniqueName: \"kubernetes.io/projected/d0e7b33d-6524-410e-90bc-25212001726b-kube-api-access-xzjcz\") pod \"test-model-00d870-kserve-7b75fd6955-lhdgf\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.589585 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.589530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:26:55.716230 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.716200 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf"] Jul 16 09:26:55.719434 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:26:55.719407 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0e7b33d_6524_410e_90bc_25212001726b.slice/crio-c3a0ae190b99a6b5b5a5d3a5e794378e4c5a2d516614ea24fc81a55c304ab8a1 WatchSource:0}: Error finding container c3a0ae190b99a6b5b5a5d3a5e794378e4c5a2d516614ea24fc81a55c304ab8a1: Status 404 returned error can't find the container with id c3a0ae190b99a6b5b5a5d3a5e794378e4c5a2d516614ea24fc81a55c304ab8a1 Jul 16 09:26:55.730462 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:55.730433 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" event={"ID":"d0e7b33d-6524-410e-90bc-25212001726b","Type":"ContainerStarted","Data":"c3a0ae190b99a6b5b5a5d3a5e794378e4c5a2d516614ea24fc81a55c304ab8a1"} Jul 16 09:26:56.737026 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:56.736990 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" event={"ID":"d0e7b33d-6524-410e-90bc-25212001726b","Type":"ContainerStarted","Data":"b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3"} Jul 16 09:26:56.758084 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:56.758025 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" podStartSLOduration=1.758009929 podStartE2EDuration="1.758009929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:26:56.755584482 +0000 UTC m=+1930.129718769" watchObservedRunningTime="2026-07-16 09:26:56.758009929 +0000 UTC m=+1930.132144218" Jul 16 09:26:58.724355 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:26:58.724317 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:27:05.590603 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:05.590538 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:27:05.591035 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:05.590627 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:27:05.603076 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:05.603043 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:27:05.789699 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:05.789667 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:27:09.716617 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:09.716573 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:27:22.237924 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.237893 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx"] Jul 16 09:27:22.240630 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.240610 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.243948 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.243923 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cfc611\"/\"default-dockercfg-b5t49\"" Jul 16 09:27:22.244071 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.243974 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cfc611\"/\"openshift-service-ca.crt\"" Jul 16 09:27:22.244071 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.244024 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cfc611\"/\"kube-root-ca.crt\"" Jul 16 09:27:22.244071 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.244040 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cfc611\"/\"test-model-cfc611-kserve-self-signed-certs\"" Jul 16 09:27:22.251441 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.251421 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx"] Jul 16 09:27:22.278287 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.278260 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-model-cache\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.278385 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.278302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-home\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.278385 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.278319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-dshm\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.278456 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.278416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32694bd0-903d-4069-a141-b567baf8e605-tls-certs\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.278497 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.278451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdct2\" (UniqueName: \"kubernetes.io/projected/32694bd0-903d-4069-a141-b567baf8e605-kube-api-access-vdct2\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.278497 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.278472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-tmp-dir\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.379141 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.379110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32694bd0-903d-4069-a141-b567baf8e605-tls-certs\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.379242 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.379164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdct2\" (UniqueName: \"kubernetes.io/projected/32694bd0-903d-4069-a141-b567baf8e605-kube-api-access-vdct2\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.379242 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.379184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-tmp-dir\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.379242 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.379228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-model-cache\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.379354 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.379255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-home\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.379396 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.379368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-dshm\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.379618 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.379597 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-home\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.379684 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.379647 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-tmp-dir\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.379724 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.379707 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-model-cache\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.381567 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.381543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-dshm\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.381814 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.381793 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32694bd0-903d-4069-a141-b567baf8e605-tls-certs\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.387724 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.387702 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdct2\" (UniqueName: \"kubernetes.io/projected/32694bd0-903d-4069-a141-b567baf8e605-kube-api-access-vdct2\") pod \"test-model-cfc611-kserve-d7c66b469-g68rx\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.551644 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.551580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:22.675053 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.675027 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx"] Jul 16 09:27:22.677608 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:27:22.677580 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32694bd0_903d_4069_a141_b567baf8e605.slice/crio-73f239e74cbb61dbcf5412a868700008b37135510a2694693699453cd6c23723 WatchSource:0}: Error finding container 73f239e74cbb61dbcf5412a868700008b37135510a2694693699453cd6c23723: Status 404 returned error can't find the container with id 73f239e74cbb61dbcf5412a868700008b37135510a2694693699453cd6c23723 Jul 16 09:27:22.857022 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.856949 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" event={"ID":"32694bd0-903d-4069-a141-b567baf8e605","Type":"ContainerStarted","Data":"c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8"} Jul 16 09:27:22.857022 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.856995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" event={"ID":"32694bd0-903d-4069-a141-b567baf8e605","Type":"ContainerStarted","Data":"73f239e74cbb61dbcf5412a868700008b37135510a2694693699453cd6c23723"} Jul 16 09:27:22.875218 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:22.875170 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" podStartSLOduration=0.875139872 podStartE2EDuration="875.139872ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:27:22.874418682 +0000 UTC m=+1956.248552987" watchObservedRunningTime="2026-07-16 09:27:22.875139872 +0000 UTC m=+1956.249274161" Jul 16 09:27:26.421784 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:26.421751 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:27:32.551874 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:32.551838 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:32.552387 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:32.551896 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:32.564396 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:32.564373 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:32.913553 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:32.913528 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:40.351175 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:40.351078 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:27:41.240895 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:41.240850 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf"] Jul 16 09:27:42.291613 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:42.291581 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx"] Jul 16 09:27:53.526859 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:53.526824 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:27:56.261731 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.261667 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" podUID="d0e7b33d-6524-410e-90bc-25212001726b" containerName="main" containerID="cri-o://b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3" gracePeriod=45 Jul 16 09:27:56.504484 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.504462 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:27:56.649691 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.649666 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-model-cache\") pod \"d0e7b33d-6524-410e-90bc-25212001726b\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " Jul 16 09:27:56.649819 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.649730 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-tmp-dir\") pod \"d0e7b33d-6524-410e-90bc-25212001726b\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " Jul 16 09:27:56.649819 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.649757 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e7b33d-6524-410e-90bc-25212001726b-tls-certs\") pod \"d0e7b33d-6524-410e-90bc-25212001726b\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " Jul 16 09:27:56.649819 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.649804 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-dshm\") pod \"d0e7b33d-6524-410e-90bc-25212001726b\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " Jul 16 09:27:56.649989 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.649923 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzjcz\" (UniqueName: \"kubernetes.io/projected/d0e7b33d-6524-410e-90bc-25212001726b-kube-api-access-xzjcz\") pod \"d0e7b33d-6524-410e-90bc-25212001726b\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " Jul 16 09:27:56.649989 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.649948 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0e7b33d-6524-410e-90bc-25212001726b" (UID: "d0e7b33d-6524-410e-90bc-25212001726b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:27:56.649989 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.649973 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-home\") pod \"d0e7b33d-6524-410e-90bc-25212001726b\" (UID: \"d0e7b33d-6524-410e-90bc-25212001726b\") " Jul 16 09:27:56.650113 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.650006 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0e7b33d-6524-410e-90bc-25212001726b" (UID: "d0e7b33d-6524-410e-90bc-25212001726b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:27:56.650178 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.650139 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-model-cache\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:56.650178 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.650172 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-tmp-dir\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:56.650264 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.650245 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-home" (OuterVolumeSpecName: "home") pod "d0e7b33d-6524-410e-90bc-25212001726b" (UID: "d0e7b33d-6524-410e-90bc-25212001726b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:27:56.652069 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.652040 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-dshm" (OuterVolumeSpecName: "dshm") pod "d0e7b33d-6524-410e-90bc-25212001726b" (UID: "d0e7b33d-6524-410e-90bc-25212001726b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:27:56.652188 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.652062 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e7b33d-6524-410e-90bc-25212001726b-kube-api-access-xzjcz" (OuterVolumeSpecName: "kube-api-access-xzjcz") pod "d0e7b33d-6524-410e-90bc-25212001726b" (UID: "d0e7b33d-6524-410e-90bc-25212001726b"). InnerVolumeSpecName "kube-api-access-xzjcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:27:56.652188 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.652098 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e7b33d-6524-410e-90bc-25212001726b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0e7b33d-6524-410e-90bc-25212001726b" (UID: "d0e7b33d-6524-410e-90bc-25212001726b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 09:27:56.750675 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.750655 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0e7b33d-6524-410e-90bc-25212001726b-tls-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:56.750675 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.750675 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-dshm\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:56.750805 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.750686 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzjcz\" (UniqueName: \"kubernetes.io/projected/d0e7b33d-6524-410e-90bc-25212001726b-kube-api-access-xzjcz\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:56.750805 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:56.750694 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0e7b33d-6524-410e-90bc-25212001726b-home\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:57.028371 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.028296 2574 generic.go:356] "Generic (PLEG): container finished" podID="d0e7b33d-6524-410e-90bc-25212001726b" containerID="b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3" exitCode=0 Jul 16 09:27:57.028371 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.028341 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" Jul 16 09:27:57.028522 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.028373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" event={"ID":"d0e7b33d-6524-410e-90bc-25212001726b","Type":"ContainerDied","Data":"b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3"} Jul 16 09:27:57.028522 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.028422 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf" event={"ID":"d0e7b33d-6524-410e-90bc-25212001726b","Type":"ContainerDied","Data":"c3a0ae190b99a6b5b5a5d3a5e794378e4c5a2d516614ea24fc81a55c304ab8a1"} Jul 16 09:27:57.028522 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.028444 2574 scope.go:117] "RemoveContainer" containerID="b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3" Jul 16 09:27:57.041312 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.041284 2574 scope.go:117] "RemoveContainer" containerID="b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3" Jul 16 09:27:57.041735 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:27:57.041706 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3\": container with ID starting with b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3 not found: ID does not exist" containerID="b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3" Jul 16 09:27:57.042002 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.041832 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3"} err="failed to get container status \"b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3\": rpc error: code = NotFound desc = could not find container \"b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3\": container with ID starting with b16f09b3a2dbe09c7552491e1497aa79938775ecb37a83b9409b908233f65eb3 not found: ID does not exist" Jul 16 09:27:57.058548 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.058524 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf"] Jul 16 09:27:57.058548 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.058551 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-00d870/test-model-00d870-kserve-7b75fd6955-lhdgf"] Jul 16 09:27:57.311190 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.311103 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" podUID="32694bd0-903d-4069-a141-b567baf8e605" containerName="main" containerID="cri-o://c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8" gracePeriod=45 Jul 16 09:27:57.559320 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.559296 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:57.657261 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657236 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-home\") pod \"32694bd0-903d-4069-a141-b567baf8e605\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " Jul 16 09:27:57.657379 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657272 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32694bd0-903d-4069-a141-b567baf8e605-tls-certs\") pod \"32694bd0-903d-4069-a141-b567baf8e605\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " Jul 16 09:27:57.657379 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657320 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-tmp-dir\") pod \"32694bd0-903d-4069-a141-b567baf8e605\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " Jul 16 09:27:57.657379 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657363 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-model-cache\") pod \"32694bd0-903d-4069-a141-b567baf8e605\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " Jul 16 09:27:57.657379 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657377 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-dshm\") pod \"32694bd0-903d-4069-a141-b567baf8e605\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " Jul 16 09:27:57.657600 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657413 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdct2\" (UniqueName: \"kubernetes.io/projected/32694bd0-903d-4069-a141-b567baf8e605-kube-api-access-vdct2\") pod \"32694bd0-903d-4069-a141-b567baf8e605\" (UID: \"32694bd0-903d-4069-a141-b567baf8e605\") " Jul 16 09:27:57.657600 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657481 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-home" (OuterVolumeSpecName: "home") pod "32694bd0-903d-4069-a141-b567baf8e605" (UID: "32694bd0-903d-4069-a141-b567baf8e605"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:27:57.657708 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657621 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32694bd0-903d-4069-a141-b567baf8e605" (UID: "32694bd0-903d-4069-a141-b567baf8e605"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:27:57.657708 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657636 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-model-cache" (OuterVolumeSpecName: "model-cache") pod "32694bd0-903d-4069-a141-b567baf8e605" (UID: "32694bd0-903d-4069-a141-b567baf8e605"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:27:57.657863 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657727 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-tmp-dir\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:57.657863 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657747 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-model-cache\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:57.657863 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.657761 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-home\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:57.659564 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.659534 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-dshm" (OuterVolumeSpecName: "dshm") pod "32694bd0-903d-4069-a141-b567baf8e605" (UID: "32694bd0-903d-4069-a141-b567baf8e605"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:27:57.659677 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.659584 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32694bd0-903d-4069-a141-b567baf8e605-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32694bd0-903d-4069-a141-b567baf8e605" (UID: "32694bd0-903d-4069-a141-b567baf8e605"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 09:27:57.659677 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.659661 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32694bd0-903d-4069-a141-b567baf8e605-kube-api-access-vdct2" (OuterVolumeSpecName: "kube-api-access-vdct2") pod "32694bd0-903d-4069-a141-b567baf8e605" (UID: "32694bd0-903d-4069-a141-b567baf8e605"). InnerVolumeSpecName "kube-api-access-vdct2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:27:57.758465 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.758439 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32694bd0-903d-4069-a141-b567baf8e605-tls-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:57.758465 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.758463 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32694bd0-903d-4069-a141-b567baf8e605-dshm\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:57.758591 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:57.758474 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdct2\" (UniqueName: \"kubernetes.io/projected/32694bd0-903d-4069-a141-b567baf8e605-kube-api-access-vdct2\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:27:58.033999 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:58.033925 2574 generic.go:356] "Generic (PLEG): container finished" podID="32694bd0-903d-4069-a141-b567baf8e605" containerID="c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8" exitCode=0 Jul 16 09:27:58.033999 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:58.033973 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" Jul 16 09:27:58.034218 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:58.034016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" event={"ID":"32694bd0-903d-4069-a141-b567baf8e605","Type":"ContainerDied","Data":"c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8"} Jul 16 09:27:58.034218 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:58.034060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx" event={"ID":"32694bd0-903d-4069-a141-b567baf8e605","Type":"ContainerDied","Data":"73f239e74cbb61dbcf5412a868700008b37135510a2694693699453cd6c23723"} Jul 16 09:27:58.034218 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:58.034084 2574 scope.go:117] "RemoveContainer" containerID="c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8" Jul 16 09:27:58.042755 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:58.042738 2574 scope.go:117] "RemoveContainer" containerID="c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8" Jul 16 09:27:58.043045 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:27:58.043027 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8\": container with ID starting with c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8 not found: ID does not exist" containerID="c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8" Jul 16 09:27:58.043110 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:58.043055 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8"} err="failed to get container status \"c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8\": rpc error: code = NotFound desc = could not find container \"c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8\": container with ID starting with c3487d7adbd4940713f074f23808bf338088aba08842e795c8c6299424fb75f8 not found: ID does not exist" Jul 16 09:27:58.058338 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:58.058310 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx"] Jul 16 09:27:58.067797 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:58.067770 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cfc611/test-model-cfc611-kserve-d7c66b469-g68rx"] Jul 16 09:27:59.054681 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:59.054643 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32694bd0-903d-4069-a141-b567baf8e605" path="/var/lib/kubelet/pods/32694bd0-903d-4069-a141-b567baf8e605/volumes" Jul 16 09:27:59.055205 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:27:59.055184 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e7b33d-6524-410e-90bc-25212001726b" path="/var/lib/kubelet/pods/d0e7b33d-6524-410e-90bc-25212001726b/volumes" Jul 16 09:28:27.817372 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:27.817341 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:28:31.190813 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:31.190777 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb"] Jul 16 09:28:31.191226 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:31.190986 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" containerID="cri-o://6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987" gracePeriod=30 Jul 16 09:28:36.440358 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.440335 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:28:36.572015 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.571955 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-certs\") pod \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " Jul 16 09:28:36.572015 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572009 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-socket\") pod \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " Jul 16 09:28:36.572223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572052 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-podinfo\") pod \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " Jul 16 09:28:36.572223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572072 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-credential-socket\") pod \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " Jul 16 09:28:36.572223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572091 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26cwp\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-kube-api-access-26cwp\") pod \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " Jul 16 09:28:36.572223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572119 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-data\") pod \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " Jul 16 09:28:36.572223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572139 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-envoy\") pod \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " Jul 16 09:28:36.572223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572207 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istiod-ca-cert\") pod \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " Jul 16 09:28:36.572527 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572226 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "98f59953-ffd8-40b9-85ce-693fdbb3fb4a" (UID: "98f59953-ffd8-40b9-85ce-693fdbb3fb4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:28:36.572527 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572244 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-token\") pod \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\" (UID: \"98f59953-ffd8-40b9-85ce-693fdbb3fb4a\") " Jul 16 09:28:36.572527 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572357 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "98f59953-ffd8-40b9-85ce-693fdbb3fb4a" (UID: "98f59953-ffd8-40b9-85ce-693fdbb3fb4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:28:36.572527 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572368 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "98f59953-ffd8-40b9-85ce-693fdbb3fb4a" (UID: "98f59953-ffd8-40b9-85ce-693fdbb3fb4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:28:36.572729 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572622 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:28:36.572729 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572645 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-workload-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:28:36.572729 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572663 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-credential-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:28:36.572729 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.572659 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "98f59953-ffd8-40b9-85ce-693fdbb3fb4a" (UID: "98f59953-ffd8-40b9-85ce-693fdbb3fb4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 09:28:36.573476 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.573447 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "98f59953-ffd8-40b9-85ce-693fdbb3fb4a" (UID: "98f59953-ffd8-40b9-85ce-693fdbb3fb4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:28:36.574368 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.574340 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "98f59953-ffd8-40b9-85ce-693fdbb3fb4a" (UID: "98f59953-ffd8-40b9-85ce-693fdbb3fb4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 09:28:36.574597 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.574571 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-kube-api-access-26cwp" (OuterVolumeSpecName: "kube-api-access-26cwp") pod "98f59953-ffd8-40b9-85ce-693fdbb3fb4a" (UID: "98f59953-ffd8-40b9-85ce-693fdbb3fb4a"). InnerVolumeSpecName "kube-api-access-26cwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:28:36.574847 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.574825 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "98f59953-ffd8-40b9-85ce-693fdbb3fb4a" (UID: "98f59953-ffd8-40b9-85ce-693fdbb3fb4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:28:36.574921 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.574838 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "98f59953-ffd8-40b9-85ce-693fdbb3fb4a" (UID: "98f59953-ffd8-40b9-85ce-693fdbb3fb4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:28:36.673815 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.673789 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istiod-ca-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:28:36.673815 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.673812 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:28:36.673815 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.673821 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-podinfo\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:28:36.674004 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.673830 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26cwp\" (UniqueName: \"kubernetes.io/projected/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-kube-api-access-26cwp\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:28:36.674004 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.673838 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-data\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:28:36.674004 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:36.673847 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98f59953-ffd8-40b9-85ce-693fdbb3fb4a-istio-envoy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:28:37.218377 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.218344 2574 generic.go:356] "Generic (PLEG): container finished" podID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerID="6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987" exitCode=0 Jul 16 09:28:37.218539 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.218402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" event={"ID":"98f59953-ffd8-40b9-85ce-693fdbb3fb4a","Type":"ContainerDied","Data":"6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987"} Jul 16 09:28:37.218539 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.218410 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" Jul 16 09:28:37.218539 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.218461 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb" event={"ID":"98f59953-ffd8-40b9-85ce-693fdbb3fb4a","Type":"ContainerDied","Data":"ca1ae8083059e8a1132efa43bb1632bdc61dc7404808acf0e390c04b660b6fbd"} Jul 16 09:28:37.218539 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.218496 2574 scope.go:117] "RemoveContainer" containerID="6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987" Jul 16 09:28:37.228148 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.228127 2574 scope.go:117] "RemoveContainer" containerID="81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c" Jul 16 09:28:37.236408 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.236385 2574 scope.go:117] "RemoveContainer" containerID="6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987" Jul 16 09:28:37.236735 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:28:37.236711 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987\": container with ID starting with 6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987 not found: ID does not exist" containerID="6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987" Jul 16 09:28:37.236777 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.236743 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987"} err="failed to get container status \"6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987\": rpc error: code = NotFound desc = could not find container \"6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987\": container with ID starting with 6e4268157a946c4e4075663de7e6c5f72667fc73904169a8d7e574ee13578987 not found: ID does not exist" Jul 16 09:28:37.236777 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.236761 2574 scope.go:117] "RemoveContainer" containerID="81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c" Jul 16 09:28:37.237010 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:28:37.236993 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c\": container with ID starting with 81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c not found: ID does not exist" containerID="81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c" Jul 16 09:28:37.237043 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.237013 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c"} err="failed to get container status \"81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c\": rpc error: code = NotFound desc = could not find container \"81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c\": container with ID starting with 81adf299fbd0f854d91e7cbf403f9df7418cc9880589e153ae58ef5b55874d3c not found: ID does not exist" Jul 16 09:28:37.247826 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.247801 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb"] Jul 16 09:28:37.254746 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:37.254718 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-00d870-openshift-default-f465b6c5f-5dpkb"] Jul 16 09:28:39.055497 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:39.055461 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" path="/var/lib/kubelet/pods/98f59953-ffd8-40b9-85ce-693fdbb3fb4a/volumes" Jul 16 09:28:42.233439 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:42.233400 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:28:46.375575 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.375538 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q"] Jul 16 09:28:46.376014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.375929 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32694bd0-903d-4069-a141-b567baf8e605" containerName="main" Jul 16 09:28:46.376014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.375943 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="32694bd0-903d-4069-a141-b567baf8e605" containerName="main" Jul 16 09:28:46.376014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.375955 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e7b33d-6524-410e-90bc-25212001726b" containerName="main" Jul 16 09:28:46.376014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.375961 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e7b33d-6524-410e-90bc-25212001726b" containerName="main" Jul 16 09:28:46.376014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.375971 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" Jul 16 09:28:46.376014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.375977 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" Jul 16 09:28:46.376014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.375994 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" Jul 16 09:28:46.376014 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.376002 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" Jul 16 09:28:46.376389 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.376076 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" Jul 16 09:28:46.376389 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.376092 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e7b33d-6524-410e-90bc-25212001726b" containerName="main" Jul 16 09:28:46.376389 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.376103 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="32694bd0-903d-4069-a141-b567baf8e605" containerName="main" Jul 16 09:28:46.380823 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.380800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.383757 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.383734 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6b0aee-openshift-default-dockercfg-sjg6z\"" Jul 16 09:28:46.390735 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.390709 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q"] Jul 16 09:28:46.448571 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.448547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-podinfo\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.448671 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.448576 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ecd08-83fd-41c0-b1d8-2285222689c5-istiod-ca-cert\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.448671 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.448606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-credential-socket\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.448671 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.448630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-data\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.448671 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.448653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-token\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.448802 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.448688 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-certs\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.448802 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.448730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7zz\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-kube-api-access-lg7zz\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.448802 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.448754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-envoy\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.448897 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.448804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-socket\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.549784 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.549758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-podinfo\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.549906 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.549788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ecd08-83fd-41c0-b1d8-2285222689c5-istiod-ca-cert\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.549906 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.549813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-credential-socket\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550022 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.549916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-data\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550022 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.549961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-token\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550171 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.550043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-certs\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550171 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.550082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7zz\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-kube-api-access-lg7zz\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550171 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.550110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-envoy\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550333 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.550203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-credential-socket\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550385 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.550341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-data\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550385 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.550208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-socket\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550512 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.550489 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-socket\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550593 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.550568 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ecd08-83fd-41c0-b1d8-2285222689c5-istiod-ca-cert\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.550665 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.550637 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-certs\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.552401 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.552372 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-envoy\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.552839 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.552823 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-podinfo\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.558659 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.558638 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-token\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.558851 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.558831 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7zz\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-kube-api-access-lg7zz\") pod \"e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.693913 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.693845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:46.827792 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.827753 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q"] Jul 16 09:28:46.830551 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:28:46.830524 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927ecd08_83fd_41c0_b1d8_2285222689c5.slice/crio-f9f1b8a9b1164a67fb3e3491e7d30ee5cfa25a9c7f5363f9689f4ed3852069f2 WatchSource:0}: Error finding container f9f1b8a9b1164a67fb3e3491e7d30ee5cfa25a9c7f5363f9689f4ed3852069f2: Status 404 returned error can't find the container with id f9f1b8a9b1164a67fb3e3491e7d30ee5cfa25a9c7f5363f9689f4ed3852069f2 Jul 16 09:28:46.832987 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.832951 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:28:46.833060 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.833036 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:28:46.833102 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:46.833069 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:28:47.265668 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:47.265632 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" event={"ID":"927ecd08-83fd-41c0-b1d8-2285222689c5","Type":"ContainerStarted","Data":"6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e"} Jul 16 09:28:47.265777 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:47.265696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" event={"ID":"927ecd08-83fd-41c0-b1d8-2285222689c5","Type":"ContainerStarted","Data":"f9f1b8a9b1164a67fb3e3491e7d30ee5cfa25a9c7f5363f9689f4ed3852069f2"} Jul 16 09:28:47.295697 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:47.295653 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" podStartSLOduration=1.295639902 podStartE2EDuration="1.295639902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:28:47.29324205 +0000 UTC m=+2040.667376341" watchObservedRunningTime="2026-07-16 09:28:47.295639902 +0000 UTC m=+2040.669774191" Jul 16 09:28:47.694816 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:47.694782 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:47.699600 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:47.699578 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:48.271058 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:48.271027 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:48.271939 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:48.271914 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:28:52.008044 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:28:52.008010 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:29:00.327059 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:00.327020 2574 generic.go:356] "Generic (PLEG): container finished" podID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerID="6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e" exitCode=0 Jul 16 09:29:00.327694 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:00.327102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" event={"ID":"927ecd08-83fd-41c0-b1d8-2285222689c5","Type":"ContainerDied","Data":"6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e"} Jul 16 09:29:00.327694 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:00.327677 2574 scope.go:117] "RemoveContainer" containerID="6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e" Jul 16 09:29:00.328783 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:00.328753 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:00.328849 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:00.328824 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:00.328890 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:00.328858 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:01.336972 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:01.336925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" event={"ID":"927ecd08-83fd-41c0-b1d8-2285222689c5","Type":"ContainerStarted","Data":"361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26"} Jul 16 09:29:01.694837 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:01.694799 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:29:01.696133 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:01.696101 2574 patch_prober.go:28] interesting pod/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 16 09:29:01.696306 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:01.696171 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 16 09:29:02.695232 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:02.695196 2574 patch_prober.go:28] interesting pod/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 16 09:29:02.695623 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:02.695253 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 16 09:29:03.271289 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:03.271248 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:29:03.694638 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:03.694601 2574 patch_prober.go:28] interesting pod/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 16 09:29:03.694820 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:03.694664 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 16 09:29:04.694901 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:04.694861 2574 patch_prober.go:28] interesting pod/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Jul 16 09:29:04.695400 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:04.694932 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Jul 16 09:29:05.698854 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:05.698819 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:29:05.699857 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:05.699835 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:29:06.613238 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:06.613210 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:29:22.520522 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:22.520489 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:29:23.718189 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:23.718143 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q"] Jul 16 09:29:23.718656 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:23.718379 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" containerID="cri-o://361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26" gracePeriod=30 Jul 16 09:29:28.954928 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.954904 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:29:28.998359 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998303 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ecd08-83fd-41c0-b1d8-2285222689c5-istiod-ca-cert\") pod \"927ecd08-83fd-41c0-b1d8-2285222689c5\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " Jul 16 09:29:28.998359 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998339 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-credential-socket\") pod \"927ecd08-83fd-41c0-b1d8-2285222689c5\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " Jul 16 09:29:28.998538 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998374 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-token\") pod \"927ecd08-83fd-41c0-b1d8-2285222689c5\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " Jul 16 09:29:28.998538 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998489 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-socket\") pod \"927ecd08-83fd-41c0-b1d8-2285222689c5\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " Jul 16 09:29:28.998538 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998524 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-certs\") pod \"927ecd08-83fd-41c0-b1d8-2285222689c5\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " Jul 16 09:29:28.998699 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998564 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-podinfo\") pod \"927ecd08-83fd-41c0-b1d8-2285222689c5\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " Jul 16 09:29:28.998699 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998596 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-data\") pod \"927ecd08-83fd-41c0-b1d8-2285222689c5\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " Jul 16 09:29:28.998699 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998640 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg7zz\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-kube-api-access-lg7zz\") pod \"927ecd08-83fd-41c0-b1d8-2285222689c5\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " Jul 16 09:29:28.998699 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998641 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "927ecd08-83fd-41c0-b1d8-2285222689c5" (UID: "927ecd08-83fd-41c0-b1d8-2285222689c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:29:28.998699 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998659 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-envoy\") pod \"927ecd08-83fd-41c0-b1d8-2285222689c5\" (UID: \"927ecd08-83fd-41c0-b1d8-2285222689c5\") " Jul 16 09:29:28.998699 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998669 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927ecd08-83fd-41c0-b1d8-2285222689c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "927ecd08-83fd-41c0-b1d8-2285222689c5" (UID: "927ecd08-83fd-41c0-b1d8-2285222689c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 09:29:28.999025 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998722 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "927ecd08-83fd-41c0-b1d8-2285222689c5" (UID: "927ecd08-83fd-41c0-b1d8-2285222689c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:29:28.999025 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998935 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:29:28.999025 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998954 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927ecd08-83fd-41c0-b1d8-2285222689c5-istiod-ca-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:29:28.999025 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.998968 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-credential-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:29:28.999269 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.999069 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "927ecd08-83fd-41c0-b1d8-2285222689c5" (UID: "927ecd08-83fd-41c0-b1d8-2285222689c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:29:28.999515 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:28.999491 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "927ecd08-83fd-41c0-b1d8-2285222689c5" (UID: "927ecd08-83fd-41c0-b1d8-2285222689c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:29:29.000441 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.000420 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "927ecd08-83fd-41c0-b1d8-2285222689c5" (UID: "927ecd08-83fd-41c0-b1d8-2285222689c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:29:29.000704 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.000678 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-kube-api-access-lg7zz" (OuterVolumeSpecName: "kube-api-access-lg7zz") pod "927ecd08-83fd-41c0-b1d8-2285222689c5" (UID: "927ecd08-83fd-41c0-b1d8-2285222689c5"). InnerVolumeSpecName "kube-api-access-lg7zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:29:29.000998 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.000979 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "927ecd08-83fd-41c0-b1d8-2285222689c5" (UID: "927ecd08-83fd-41c0-b1d8-2285222689c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:29:29.001053 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.000997 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "927ecd08-83fd-41c0-b1d8-2285222689c5" (UID: "927ecd08-83fd-41c0-b1d8-2285222689c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 09:29:29.100713 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.100677 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-data\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:29:29.100713 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.100714 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg7zz\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-kube-api-access-lg7zz\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:29:29.100906 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.100724 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-envoy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:29:29.100906 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.100733 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:29:29.100906 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.100744 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927ecd08-83fd-41c0-b1d8-2285222689c5-workload-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:29:29.100906 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.100755 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927ecd08-83fd-41c0-b1d8-2285222689c5-istio-podinfo\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:29:29.460051 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.460017 2574 generic.go:356] "Generic (PLEG): container finished" podID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerID="361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26" exitCode=0 Jul 16 09:29:29.460264 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.460062 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" Jul 16 09:29:29.460264 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.460084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" event={"ID":"927ecd08-83fd-41c0-b1d8-2285222689c5","Type":"ContainerDied","Data":"361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26"} Jul 16 09:29:29.460264 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.460138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q" event={"ID":"927ecd08-83fd-41c0-b1d8-2285222689c5","Type":"ContainerDied","Data":"f9f1b8a9b1164a67fb3e3491e7d30ee5cfa25a9c7f5363f9689f4ed3852069f2"} Jul 16 09:29:29.460264 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.460179 2574 scope.go:117] "RemoveContainer" containerID="361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26" Jul 16 09:29:29.468554 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.468535 2574 scope.go:117] "RemoveContainer" containerID="6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e" Jul 16 09:29:29.475864 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.475846 2574 scope.go:117] "RemoveContainer" containerID="361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26" Jul 16 09:29:29.476107 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:29:29.476088 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26\": container with ID starting with 361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26 not found: ID does not exist" containerID="361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26" Jul 16 09:29:29.476173 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.476112 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26"} err="failed to get container status \"361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26\": rpc error: code = NotFound desc = could not find container \"361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26\": container with ID starting with 361c75fdc6269f9eb757402757442fd1756758b2304c3c315a8415cd3c2edd26 not found: ID does not exist" Jul 16 09:29:29.476173 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.476127 2574 scope.go:117] "RemoveContainer" containerID="6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e" Jul 16 09:29:29.476369 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:29:29.476350 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e\": container with ID starting with 6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e not found: ID does not exist" containerID="6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e" Jul 16 09:29:29.476410 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.476375 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e"} err="failed to get container status \"6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e\": rpc error: code = NotFound desc = could not find container \"6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e\": container with ID starting with 6fb7ed7b1c20e6e51b23ca0132c919ac69c97b6fdd068496a1360b377c48dc1e not found: ID does not exist" Jul 16 09:29:29.480033 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.480013 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q"] Jul 16 09:29:29.484351 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:29.484332 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6b0aee-openshift-default-865d44c4d9-lqs7q"] Jul 16 09:29:31.055695 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:31.055655 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" path="/var/lib/kubelet/pods/927ecd08-83fd-41c0-b1d8-2285222689c5/volumes" Jul 16 09:29:32.719987 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:32.719952 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:29:37.529202 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.529171 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj"] Jul 16 09:29:37.529565 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.529478 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" Jul 16 09:29:37.529565 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.529489 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" Jul 16 09:29:37.529565 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.529499 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" Jul 16 09:29:37.529565 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.529505 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" Jul 16 09:29:37.529694 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.529577 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f59953-ffd8-40b9-85ce-693fdbb3fb4a" containerName="istio-proxy" Jul 16 09:29:37.529694 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.529587 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" Jul 16 09:29:37.529694 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.529593 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="927ecd08-83fd-41c0-b1d8-2285222689c5" containerName="istio-proxy" Jul 16 09:29:37.534018 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.533995 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.536609 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.536590 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-923c3a-openshift-default-dockercfg-8ggmv\"" Jul 16 09:29:37.546716 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.546695 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj"] Jul 16 09:29:37.661806 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.661781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-socket\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.661935 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.661814 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-certs\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.661935 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.661832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-data\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.661935 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.661849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d52fw\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-kube-api-access-d52fw\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.662082 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.661950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/524a0c18-f938-4c76-8833-ce62a32465ea-istio-podinfo\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.662082 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.661994 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-credential-socket\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.662082 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.662049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-envoy\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.662082 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.662071 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-istio-token\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.662266 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.662087 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/524a0c18-f938-4c76-8833-ce62a32465ea-istiod-ca-cert\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.762736 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.762710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/524a0c18-f938-4c76-8833-ce62a32465ea-istio-podinfo\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.762852 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.762747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-credential-socket\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.762852 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.762776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-envoy\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.762852 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.762800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-istio-token\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.762852 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.762823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/524a0c18-f938-4c76-8833-ce62a32465ea-istiod-ca-cert\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.763062 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.762859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-socket\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.763062 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.762887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-certs\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.763062 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.762912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-data\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.763062 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.762995 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d52fw\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-kube-api-access-d52fw\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.763343 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.763320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-certs\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.763343 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.763339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-socket\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.763459 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.763344 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-credential-socket\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.763519 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.763471 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-data\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.763598 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.763579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/524a0c18-f938-4c76-8833-ce62a32465ea-istiod-ca-cert\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.765321 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.765300 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-envoy\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.765420 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.765403 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/524a0c18-f938-4c76-8833-ce62a32465ea-istio-podinfo\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.771668 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.771640 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d52fw\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-kube-api-access-d52fw\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.771788 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.771724 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-istio-token\") pod \"e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.845800 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.845750 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:37.976247 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.976221 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj"] Jul 16 09:29:37.978140 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:29:37.978101 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524a0c18_f938_4c76_8833_ce62a32465ea.slice/crio-05ee81893a2d5a398fb2aabac030017aab5602fc3323a3963c1d43c65fa79f4c WatchSource:0}: Error finding container 05ee81893a2d5a398fb2aabac030017aab5602fc3323a3963c1d43c65fa79f4c: Status 404 returned error can't find the container with id 05ee81893a2d5a398fb2aabac030017aab5602fc3323a3963c1d43c65fa79f4c Jul 16 09:29:37.980179 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.980123 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:37.980270 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.980215 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:37.980270 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:37.980245 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:38.501164 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:38.501113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" event={"ID":"524a0c18-f938-4c76-8833-ce62a32465ea","Type":"ContainerStarted","Data":"872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709"} Jul 16 09:29:38.501331 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:38.501199 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" event={"ID":"524a0c18-f938-4c76-8833-ce62a32465ea","Type":"ContainerStarted","Data":"05ee81893a2d5a398fb2aabac030017aab5602fc3323a3963c1d43c65fa79f4c"} Jul 16 09:29:38.521752 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:38.521702 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" podStartSLOduration=1.5216882379999999 podStartE2EDuration="1.521688238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:29:38.521644968 +0000 UTC m=+2091.895779259" watchObservedRunningTime="2026-07-16 09:29:38.521688238 +0000 UTC m=+2091.895822528" Jul 16 09:29:38.846822 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:38.846752 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:39.850610 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:39.850578 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:40.510978 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:40.510935 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:40.511895 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:40.511874 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:42.311443 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:42.311392 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:29:45.729248 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.719368 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2"] Jul 16 09:29:45.729248 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.724028 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.729248 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.726653 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-800be8-openshift-default-dockercfg-qxv99\"" Jul 16 09:29:45.746173 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.745053 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2"] Jul 16 09:29:45.832586 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.832549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbdx\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-kube-api-access-mhbdx\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.832765 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.832607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631fb0cc-2136-4bb6-8f63-066d7584d52d-istiod-ca-cert\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.832765 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.832640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-socket\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.832765 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.832676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-token\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.832765 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.832713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-credential-socket\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.832765 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.832737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-data\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.832765 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.832764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-podinfo\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.833119 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.832807 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-certs\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.833119 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.832870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-envoy\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.934387 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.934344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbdx\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-kube-api-access-mhbdx\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.934568 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.934407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631fb0cc-2136-4bb6-8f63-066d7584d52d-istiod-ca-cert\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.934568 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.934440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-socket\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.934568 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.934480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-token\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.934568 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.934520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-credential-socket\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.934568 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.934545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-data\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.934861 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.934579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-podinfo\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.934861 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.934627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-certs\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.934861 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.934693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-envoy\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.936303 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.936277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631fb0cc-2136-4bb6-8f63-066d7584d52d-istiod-ca-cert\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.937839 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.936698 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-data\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.938034 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.936923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-credential-socket\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.938346 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.937496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-certs\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.938460 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.937792 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-socket\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.938550 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.938302 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-envoy\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.942711 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.942665 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-podinfo\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.945498 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.945472 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbdx\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-kube-api-access-mhbdx\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:45.946011 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:45.945985 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-token\") pod \"e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:46.052122 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:46.052062 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:46.243170 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:46.243118 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2"] Jul 16 09:29:46.243324 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:29:46.243298 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631fb0cc_2136_4bb6_8f63_066d7584d52d.slice/crio-5c98e368e70797e8d3db09919b392aba5c2c1d45ed9737a519e0339ea0359ac1 WatchSource:0}: Error finding container 5c98e368e70797e8d3db09919b392aba5c2c1d45ed9737a519e0339ea0359ac1: Status 404 returned error can't find the container with id 5c98e368e70797e8d3db09919b392aba5c2c1d45ed9737a519e0339ea0359ac1 Jul 16 09:29:46.246002 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:46.245515 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:46.246002 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:46.245598 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:46.246002 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:46.245641 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:46.546024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:46.545658 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" event={"ID":"631fb0cc-2136-4bb6-8f63-066d7584d52d","Type":"ContainerStarted","Data":"74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0"} Jul 16 09:29:46.546024 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:46.545722 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" event={"ID":"631fb0cc-2136-4bb6-8f63-066d7584d52d","Type":"ContainerStarted","Data":"5c98e368e70797e8d3db09919b392aba5c2c1d45ed9737a519e0339ea0359ac1"} Jul 16 09:29:46.565901 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:46.565857 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" podStartSLOduration=1.565836094 podStartE2EDuration="1.565836094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:29:46.564924831 +0000 UTC m=+2099.939059336" watchObservedRunningTime="2026-07-16 09:29:46.565836094 +0000 UTC m=+2099.939970365" Jul 16 09:29:47.056677 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:47.056644 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:47.057403 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:47.057384 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:47.156709 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:47.156678 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:29:47.158730 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:47.158711 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:29:47.550836 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:47.550808 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:47.551840 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:47.551819 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:29:50.563781 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:50.563694 2574 generic.go:356] "Generic (PLEG): container finished" podID="524a0c18-f938-4c76-8833-ce62a32465ea" containerID="872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709" exitCode=0 Jul 16 09:29:50.563781 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:50.563755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" event={"ID":"524a0c18-f938-4c76-8833-ce62a32465ea","Type":"ContainerDied","Data":"872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709"} Jul 16 09:29:50.564212 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:50.564106 2574 scope.go:117] "RemoveContainer" containerID="872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709" Jul 16 09:29:50.565093 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:50.565057 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:50.565190 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:50.565129 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:50.565190 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:50.565184 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:51.570274 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:51.570239 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" event={"ID":"524a0c18-f938-4c76-8833-ce62a32465ea","Type":"ContainerStarted","Data":"7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3"} Jul 16 09:29:51.846654 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:51.846550 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:51.847942 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:51.847908 2574 patch_prober.go:28] interesting pod/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 16 09:29:51.848086 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:51.847973 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 16 09:29:52.847029 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:52.846547 2574 patch_prober.go:28] interesting pod/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 16 09:29:52.847029 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:52.846633 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 16 09:29:53.847318 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:53.846768 2574 patch_prober.go:28] interesting pod/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 16 09:29:53.847318 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:53.846845 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 16 09:29:54.846859 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:54.846327 2574 patch_prober.go:28] interesting pod/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 16 09:29:54.846859 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:54.846400 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 16 09:29:55.511617 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:55.511572 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:55.852657 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:55.852563 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:55.853789 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:55.853765 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:29:56.911874 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:56.911825 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:29:58.604981 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:58.604945 2574 generic.go:356] "Generic (PLEG): container finished" podID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerID="74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0" exitCode=0 Jul 16 09:29:58.604981 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:58.604988 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" event={"ID":"631fb0cc-2136-4bb6-8f63-066d7584d52d","Type":"ContainerDied","Data":"74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0"} Jul 16 09:29:58.605529 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:58.605508 2574 scope.go:117] "RemoveContainer" containerID="74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0" Jul 16 09:29:58.606771 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:58.606732 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:58.606849 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:58.606817 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:58.606890 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:58.606860 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 16 09:29:59.611553 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:29:59.611517 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" event={"ID":"631fb0cc-2136-4bb6-8f63-066d7584d52d","Type":"ContainerStarted","Data":"a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c"} Jul 16 09:30:00.052850 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:00.052750 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:30:00.053965 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:00.053937 2574 patch_prober.go:28] interesting pod/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 16 09:30:00.054100 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:00.053991 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 16 09:30:01.053165 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:01.053117 2574 patch_prober.go:28] interesting pod/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 16 09:30:01.053568 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:01.053184 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 16 09:30:02.052646 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:02.052603 2574 patch_prober.go:28] interesting pod/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 16 09:30:02.052872 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:02.052681 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 16 09:30:02.550876 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:02.550839 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:30:03.052984 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:03.052944 2574 patch_prober.go:28] interesting pod/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 16 09:30:03.053198 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:03.053009 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 16 09:30:04.056453 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:04.056415 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:30:04.057356 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:04.057339 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:30:05.255891 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.255859 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z"] Jul 16 09:30:05.259487 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.259467 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.262004 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.261984 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-923c3a\"/\"openshift-service-ca.crt\"" Jul 16 09:30:05.263230 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.263203 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-923c3a\"/\"auth-test-model-923c3a-kserve-self-signed-certs\"" Jul 16 09:30:05.263338 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.263229 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-923c3a\"/\"kube-root-ca.crt\"" Jul 16 09:30:05.263338 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.263243 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-923c3a\"/\"default-dockercfg-9gjkt\"" Jul 16 09:30:05.268349 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.268328 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z"] Jul 16 09:30:05.326863 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.326836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-dshm\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.327008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.326935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llc5h\" (UniqueName: \"kubernetes.io/projected/887d8bc6-651c-43ba-a297-0303e9b9dc61-kube-api-access-llc5h\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.327008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.326978 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-model-cache\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.327008 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.327001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-tmp-dir\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.327189 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.327017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d8bc6-651c-43ba-a297-0303e9b9dc61-tls-certs\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.327189 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.327034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-home\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.427719 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.427682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-dshm\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.427914 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.427776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llc5h\" (UniqueName: \"kubernetes.io/projected/887d8bc6-651c-43ba-a297-0303e9b9dc61-kube-api-access-llc5h\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.427914 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.427819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-model-cache\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.427914 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.427848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-tmp-dir\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.427914 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.427889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d8bc6-651c-43ba-a297-0303e9b9dc61-tls-certs\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.427914 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.427912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-home\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.428242 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.428216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-model-cache\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.428306 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.428285 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-home\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.428351 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.428327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-tmp-dir\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.429885 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.429861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-dshm\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.430224 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.430208 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d8bc6-651c-43ba-a297-0303e9b9dc61-tls-certs\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.436492 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.436470 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llc5h\" (UniqueName: \"kubernetes.io/projected/887d8bc6-651c-43ba-a297-0303e9b9dc61-kube-api-access-llc5h\") pod \"auth-test-model-923c3a-kserve-669fcc58bc-tw84z\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.570063 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.569962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:05.701460 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:05.701409 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z"] Jul 16 09:30:06.663909 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:06.663870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" event={"ID":"887d8bc6-651c-43ba-a297-0303e9b9dc61","Type":"ContainerStarted","Data":"a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296"} Jul 16 09:30:06.663909 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:06.663915 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" event={"ID":"887d8bc6-651c-43ba-a297-0303e9b9dc61","Type":"ContainerStarted","Data":"5c461c89bceaaca5bd52e6515ec4ee2678bdf4675deedf76626505c6f4a9f88f"} Jul 16 09:30:06.686624 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:06.686583 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" podStartSLOduration=1.686567563 podStartE2EDuration="1.686567563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:30:06.685287878 +0000 UTC m=+2120.059422168" watchObservedRunningTime="2026-07-16 09:30:06.686567563 +0000 UTC m=+2120.060701854" Jul 16 09:30:09.407855 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:09.407817 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:30:15.570803 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:15.570764 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:15.571364 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:15.571044 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:15.583532 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:15.583507 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:15.716100 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:15.716080 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:30:30.113861 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:30.113827 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:30:44.724475 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:30:44.724376 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:31:03.815492 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:31:03.815459 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:31:18.216424 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:31:18.216378 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:31:33.112376 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:31:33.112333 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:31:48.418235 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:31:48.418197 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:32:02.814015 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:02.813978 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:32:21.308268 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:21.308183 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:32:24.896772 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:24.896731 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z"] Jul 16 09:32:35.163165 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.163040 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss"] Jul 16 09:32:35.166274 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.166253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.168695 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.168659 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-923c3a\"/\"sub-test-model-923c3a-kserve-self-signed-certs\"" Jul 16 09:32:35.174856 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.174835 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss"] Jul 16 09:32:35.249235 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.249211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-dshm\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.249323 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.249246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88acd515-1a4a-4498-979a-bb003745ddeb-tls-certs\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.249323 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.249285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-tmp-dir\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.249402 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.249322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-home\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.249402 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.249384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-model-cache\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.249466 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.249408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbsdv\" (UniqueName: \"kubernetes.io/projected/88acd515-1a4a-4498-979a-bb003745ddeb-kube-api-access-qbsdv\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.349917 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.349892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-model-cache\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.350009 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.349925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbsdv\" (UniqueName: \"kubernetes.io/projected/88acd515-1a4a-4498-979a-bb003745ddeb-kube-api-access-qbsdv\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.350009 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.349945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-dshm\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.350009 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.349974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88acd515-1a4a-4498-979a-bb003745ddeb-tls-certs\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.350184 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.350033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-tmp-dir\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.350184 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.350058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-home\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.350415 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.350389 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-model-cache\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.350492 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.350415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-home\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.350549 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.350492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-tmp-dir\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.352347 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.352321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-dshm\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.352535 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.352516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88acd515-1a4a-4498-979a-bb003745ddeb-tls-certs\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.358861 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.358833 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbsdv\" (UniqueName: \"kubernetes.io/projected/88acd515-1a4a-4498-979a-bb003745ddeb-kube-api-access-qbsdv\") pod \"sub-test-model-923c3a-kserve-569bc7f89-jwhss\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.475199 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.475099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:35.616685 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:35.616660 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss"] Jul 16 09:32:35.618853 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:32:35.618817 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88acd515_1a4a_4498_979a_bb003745ddeb.slice/crio-2232bfffd4a3670a4d0b29365bf764d98c445700ff2a6f1c82b40455ef0ebd9a WatchSource:0}: Error finding container 2232bfffd4a3670a4d0b29365bf764d98c445700ff2a6f1c82b40455ef0ebd9a: Status 404 returned error can't find the container with id 2232bfffd4a3670a4d0b29365bf764d98c445700ff2a6f1c82b40455ef0ebd9a Jul 16 09:32:36.342661 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:36.342624 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" event={"ID":"88acd515-1a4a-4498-979a-bb003745ddeb","Type":"ContainerStarted","Data":"24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99"} Jul 16 09:32:36.343018 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:36.342669 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" event={"ID":"88acd515-1a4a-4498-979a-bb003745ddeb","Type":"ContainerStarted","Data":"2232bfffd4a3670a4d0b29365bf764d98c445700ff2a6f1c82b40455ef0ebd9a"} Jul 16 09:32:36.359721 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:36.359679 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" podStartSLOduration=1.359664917 podStartE2EDuration="1.359664917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:32:36.358579508 +0000 UTC m=+2269.732713798" watchObservedRunningTime="2026-07-16 09:32:36.359664917 +0000 UTC m=+2269.733799185" Jul 16 09:32:38.821803 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:38.821763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:32:39.916329 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:39.916283 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" podUID="887d8bc6-651c-43ba-a297-0303e9b9dc61" containerName="main" containerID="cri-o://a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296" gracePeriod=45 Jul 16 09:32:40.159620 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.159594 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:32:40.291451 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.291380 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-tmp-dir\") pod \"887d8bc6-651c-43ba-a297-0303e9b9dc61\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " Jul 16 09:32:40.291451 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.291414 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d8bc6-651c-43ba-a297-0303e9b9dc61-tls-certs\") pod \"887d8bc6-651c-43ba-a297-0303e9b9dc61\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " Jul 16 09:32:40.291640 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.291453 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llc5h\" (UniqueName: \"kubernetes.io/projected/887d8bc6-651c-43ba-a297-0303e9b9dc61-kube-api-access-llc5h\") pod \"887d8bc6-651c-43ba-a297-0303e9b9dc61\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " Jul 16 09:32:40.291640 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.291482 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-model-cache\") pod \"887d8bc6-651c-43ba-a297-0303e9b9dc61\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " Jul 16 09:32:40.291640 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.291501 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-home\") pod \"887d8bc6-651c-43ba-a297-0303e9b9dc61\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " Jul 16 09:32:40.291640 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.291527 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-dshm\") pod \"887d8bc6-651c-43ba-a297-0303e9b9dc61\" (UID: \"887d8bc6-651c-43ba-a297-0303e9b9dc61\") " Jul 16 09:32:40.291827 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.291702 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "887d8bc6-651c-43ba-a297-0303e9b9dc61" (UID: "887d8bc6-651c-43ba-a297-0303e9b9dc61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:32:40.291885 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.291828 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-home" (OuterVolumeSpecName: "home") pod "887d8bc6-651c-43ba-a297-0303e9b9dc61" (UID: "887d8bc6-651c-43ba-a297-0303e9b9dc61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:32:40.291885 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.291850 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-model-cache" (OuterVolumeSpecName: "model-cache") pod "887d8bc6-651c-43ba-a297-0303e9b9dc61" (UID: "887d8bc6-651c-43ba-a297-0303e9b9dc61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:32:40.293716 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.293692 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887d8bc6-651c-43ba-a297-0303e9b9dc61-kube-api-access-llc5h" (OuterVolumeSpecName: "kube-api-access-llc5h") pod "887d8bc6-651c-43ba-a297-0303e9b9dc61" (UID: "887d8bc6-651c-43ba-a297-0303e9b9dc61"). InnerVolumeSpecName "kube-api-access-llc5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:32:40.293716 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.293707 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887d8bc6-651c-43ba-a297-0303e9b9dc61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "887d8bc6-651c-43ba-a297-0303e9b9dc61" (UID: "887d8bc6-651c-43ba-a297-0303e9b9dc61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 09:32:40.293868 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.293786 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-dshm" (OuterVolumeSpecName: "dshm") pod "887d8bc6-651c-43ba-a297-0303e9b9dc61" (UID: "887d8bc6-651c-43ba-a297-0303e9b9dc61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:32:40.361901 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.361878 2574 generic.go:356] "Generic (PLEG): container finished" podID="887d8bc6-651c-43ba-a297-0303e9b9dc61" containerID="a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296" exitCode=0 Jul 16 09:32:40.362007 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.361919 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" Jul 16 09:32:40.362069 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.361920 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" event={"ID":"887d8bc6-651c-43ba-a297-0303e9b9dc61","Type":"ContainerDied","Data":"a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296"} Jul 16 09:32:40.362069 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.362058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z" event={"ID":"887d8bc6-651c-43ba-a297-0303e9b9dc61","Type":"ContainerDied","Data":"5c461c89bceaaca5bd52e6515ec4ee2678bdf4675deedf76626505c6f4a9f88f"} Jul 16 09:32:40.362219 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.362091 2574 scope.go:117] "RemoveContainer" containerID="a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296" Jul 16 09:32:40.370737 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.370719 2574 scope.go:117] "RemoveContainer" containerID="a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296" Jul 16 09:32:40.370994 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:32:40.370970 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296\": container with ID starting with a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296 not found: ID does not exist" containerID="a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296" Jul 16 09:32:40.371062 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.371001 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296"} err="failed to get container status \"a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296\": rpc error: code = NotFound desc = could not find container \"a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296\": container with ID starting with a960d082137bbcb0442d2191b12206f1bd9078d7be799c94d6f56f3786a99296 not found: ID does not exist" Jul 16 09:32:40.385603 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.385580 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z"] Jul 16 09:32:40.392214 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.392195 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/auth-test-model-923c3a-kserve-669fcc58bc-tw84z"] Jul 16 09:32:40.392501 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.392481 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-tmp-dir\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:32:40.392544 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.392506 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887d8bc6-651c-43ba-a297-0303e9b9dc61-tls-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:32:40.392544 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.392522 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llc5h\" (UniqueName: \"kubernetes.io/projected/887d8bc6-651c-43ba-a297-0303e9b9dc61-kube-api-access-llc5h\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:32:40.392544 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.392530 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-model-cache\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:32:40.392544 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.392538 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-home\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:32:40.392669 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:40.392545 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887d8bc6-651c-43ba-a297-0303e9b9dc61-dshm\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:32:41.055407 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:41.055371 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="887d8bc6-651c-43ba-a297-0303e9b9dc61" path="/var/lib/kubelet/pods/887d8bc6-651c-43ba-a297-0303e9b9dc61/volumes" Jul 16 09:32:45.475476 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:45.475382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:45.475863 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:45.475492 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:45.488207 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:45.488182 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:32:46.406947 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:32:46.406917 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:33:17.138605 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:17.138574 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:33:37.824249 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:37.824136 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:33:52.151204 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:52.151131 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss"] Jul 16 09:33:57.308547 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:57.308500 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:33:59.359304 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.359261 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv"] Jul 16 09:33:59.359706 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.359651 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887d8bc6-651c-43ba-a297-0303e9b9dc61" containerName="main" Jul 16 09:33:59.359706 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.359664 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="887d8bc6-651c-43ba-a297-0303e9b9dc61" containerName="main" Jul 16 09:33:59.359781 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.359739 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="887d8bc6-651c-43ba-a297-0303e9b9dc61" containerName="main" Jul 16 09:33:59.362896 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.362874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.365498 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.365472 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-923c3a\"/\"rate-test-model-923c3a-kserve-self-signed-certs\"" Jul 16 09:33:59.372619 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.372584 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv"] Jul 16 09:33:59.441843 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.441775 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-home\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.441843 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.441844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-dshm\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.442101 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.441871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swwdq\" (UniqueName: \"kubernetes.io/projected/671090cf-5723-418e-b055-b0c8c5c81eb9-kube-api-access-swwdq\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.442101 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.442012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-model-cache\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.442101 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.442087 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671090cf-5723-418e-b055-b0c8c5c81eb9-tls-certs\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.442234 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.442126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-tmp-dir\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.543531 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.543487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-dshm\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.543531 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.543533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swwdq\" (UniqueName: \"kubernetes.io/projected/671090cf-5723-418e-b055-b0c8c5c81eb9-kube-api-access-swwdq\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.543785 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.543610 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-model-cache\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.543785 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.543759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671090cf-5723-418e-b055-b0c8c5c81eb9-tls-certs\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.543918 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.543805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-tmp-dir\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.543918 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.543878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-home\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.544036 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.544012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-model-cache\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.544196 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.544176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-tmp-dir\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.544283 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.544241 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-home\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.546092 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.546067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-dshm\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.546351 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.546330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671090cf-5723-418e-b055-b0c8c5c81eb9-tls-certs\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.552467 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.552439 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swwdq\" (UniqueName: \"kubernetes.io/projected/671090cf-5723-418e-b055-b0c8c5c81eb9-kube-api-access-swwdq\") pod \"rate-test-model-923c3a-kserve-75864884f5-872jv\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.673438 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.673381 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:33:59.819296 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:33:59.819261 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv"] Jul 16 09:33:59.821465 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:33:59.821425 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod671090cf_5723_418e_b055_b0c8c5c81eb9.slice/crio-e4e4a16edb598ca783e50fcb91d9b4e2e54a8508edd4c4affafa8d5d0703ddd4 WatchSource:0}: Error finding container e4e4a16edb598ca783e50fcb91d9b4e2e54a8508edd4c4affafa8d5d0703ddd4: Status 404 returned error can't find the container with id e4e4a16edb598ca783e50fcb91d9b4e2e54a8508edd4c4affafa8d5d0703ddd4 Jul 16 09:34:00.742567 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:00.742519 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" event={"ID":"671090cf-5723-418e-b055-b0c8c5c81eb9","Type":"ContainerStarted","Data":"3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb"} Jul 16 09:34:00.742567 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:00.742576 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" event={"ID":"671090cf-5723-418e-b055-b0c8c5c81eb9","Type":"ContainerStarted","Data":"e4e4a16edb598ca783e50fcb91d9b4e2e54a8508edd4c4affafa8d5d0703ddd4"} Jul 16 09:34:00.761003 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:00.760947 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" podStartSLOduration=1.760926466 podStartE2EDuration="1.760926466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:34:00.758946149 +0000 UTC m=+2354.133080454" watchObservedRunningTime="2026-07-16 09:34:00.760926466 +0000 UTC m=+2354.135060787" Jul 16 09:34:07.171321 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.171280 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" podUID="88acd515-1a4a-4498-979a-bb003745ddeb" containerName="main" containerID="cri-o://24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99" gracePeriod=45 Jul 16 09:34:07.420344 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.420311 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:34:07.516679 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.516589 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-tmp-dir\") pod \"88acd515-1a4a-4498-979a-bb003745ddeb\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " Jul 16 09:34:07.516679 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.516638 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-dshm\") pod \"88acd515-1a4a-4498-979a-bb003745ddeb\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " Jul 16 09:34:07.516679 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.516673 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-model-cache\") pod \"88acd515-1a4a-4498-979a-bb003745ddeb\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " Jul 16 09:34:07.516968 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.516725 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-home\") pod \"88acd515-1a4a-4498-979a-bb003745ddeb\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " Jul 16 09:34:07.516968 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.516765 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88acd515-1a4a-4498-979a-bb003745ddeb-tls-certs\") pod \"88acd515-1a4a-4498-979a-bb003745ddeb\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " Jul 16 09:34:07.516968 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.516843 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbsdv\" (UniqueName: \"kubernetes.io/projected/88acd515-1a4a-4498-979a-bb003745ddeb-kube-api-access-qbsdv\") pod \"88acd515-1a4a-4498-979a-bb003745ddeb\" (UID: \"88acd515-1a4a-4498-979a-bb003745ddeb\") " Jul 16 09:34:07.516968 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.516920 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88acd515-1a4a-4498-979a-bb003745ddeb" (UID: "88acd515-1a4a-4498-979a-bb003745ddeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:34:07.517199 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.516963 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-model-cache" (OuterVolumeSpecName: "model-cache") pod "88acd515-1a4a-4498-979a-bb003745ddeb" (UID: "88acd515-1a4a-4498-979a-bb003745ddeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:34:07.517199 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.517006 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-home" (OuterVolumeSpecName: "home") pod "88acd515-1a4a-4498-979a-bb003745ddeb" (UID: "88acd515-1a4a-4498-979a-bb003745ddeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:34:07.517199 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.517100 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-tmp-dir\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:34:07.517199 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.517122 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-model-cache\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:34:07.517199 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.517136 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-home\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:34:07.518921 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.518899 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88acd515-1a4a-4498-979a-bb003745ddeb-kube-api-access-qbsdv" (OuterVolumeSpecName: "kube-api-access-qbsdv") pod "88acd515-1a4a-4498-979a-bb003745ddeb" (UID: "88acd515-1a4a-4498-979a-bb003745ddeb"). InnerVolumeSpecName "kube-api-access-qbsdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:34:07.518991 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.518934 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88acd515-1a4a-4498-979a-bb003745ddeb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88acd515-1a4a-4498-979a-bb003745ddeb" (UID: "88acd515-1a4a-4498-979a-bb003745ddeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 09:34:07.518991 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.518976 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-dshm" (OuterVolumeSpecName: "dshm") pod "88acd515-1a4a-4498-979a-bb003745ddeb" (UID: "88acd515-1a4a-4498-979a-bb003745ddeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:34:07.617586 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.617547 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88acd515-1a4a-4498-979a-bb003745ddeb-tls-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:34:07.617586 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.617582 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbsdv\" (UniqueName: \"kubernetes.io/projected/88acd515-1a4a-4498-979a-bb003745ddeb-kube-api-access-qbsdv\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:34:07.617814 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.617595 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88acd515-1a4a-4498-979a-bb003745ddeb-dshm\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:34:07.779113 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.779028 2574 generic.go:356] "Generic (PLEG): container finished" podID="88acd515-1a4a-4498-979a-bb003745ddeb" containerID="24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99" exitCode=0 Jul 16 09:34:07.779113 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.779072 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" Jul 16 09:34:07.779113 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.779085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" event={"ID":"88acd515-1a4a-4498-979a-bb003745ddeb","Type":"ContainerDied","Data":"24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99"} Jul 16 09:34:07.779393 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.779116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss" event={"ID":"88acd515-1a4a-4498-979a-bb003745ddeb","Type":"ContainerDied","Data":"2232bfffd4a3670a4d0b29365bf764d98c445700ff2a6f1c82b40455ef0ebd9a"} Jul 16 09:34:07.779393 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.779134 2574 scope.go:117] "RemoveContainer" containerID="24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99" Jul 16 09:34:07.787918 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.787898 2574 scope.go:117] "RemoveContainer" containerID="24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99" Jul 16 09:34:07.788195 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:34:07.788149 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99\": container with ID starting with 24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99 not found: ID does not exist" containerID="24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99" Jul 16 09:34:07.788266 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.788209 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99"} err="failed to get container status \"24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99\": rpc error: code = NotFound desc = could not find container \"24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99\": container with ID starting with 24387fc1aae98dd274bd3591896c7ed9a165470b0ad0eaaa6912f2e91cd2fc99 not found: ID does not exist" Jul 16 09:34:07.800573 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.800547 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss"] Jul 16 09:34:07.804575 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:07.804554 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/sub-test-model-923c3a-kserve-569bc7f89-jwhss"] Jul 16 09:34:09.062309 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:09.062253 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88acd515-1a4a-4498-979a-bb003745ddeb" path="/var/lib/kubelet/pods/88acd515-1a4a-4498-979a-bb003745ddeb/volumes" Jul 16 09:34:09.674172 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:09.674107 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:34:09.674379 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:09.674198 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:34:09.686391 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:09.686365 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:34:09.799466 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:09.799431 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:34:09.810787 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:09.810759 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:34:47.183491 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:47.183458 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:34:47.186447 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:47.186428 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lf9r4_08a75a91-2b97-4b16-8951-8d91c167dea5/ovn-acl-logging/0.log" Jul 16 09:34:51.313386 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:34:51.313347 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:35:06.820716 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:06.820682 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:35:22.817055 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:22.817012 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:35:28.333700 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:28.333668 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv"] Jul 16 09:35:30.352575 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.352533 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7753-140460068331952"] Jul 16 09:35:30.353124 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.352881 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88acd515-1a4a-4498-979a-bb003745ddeb" containerName="main" Jul 16 09:35:30.353124 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.352893 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="88acd515-1a4a-4498-979a-bb003745ddeb" containerName="main" Jul 16 09:35:30.353124 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.352953 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="88acd515-1a4a-4498-979a-bb003745ddeb" containerName="main" Jul 16 09:35:30.356121 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.356101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460068331952" Jul 16 09:35:30.359051 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.359014 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5x545\"" Jul 16 09:35:30.360128 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.360100 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460068331952"] Jul 16 09:35:30.517735 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.517715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxxbt\" (UniqueName: \"kubernetes.io/projected/f3539ebf-c534-4247-b9b3-3b5ce19b366b-kube-api-access-xxxbt\") pod \"test-curl-7753-140460068331952\" (UID: \"f3539ebf-c534-4247-b9b3-3b5ce19b366b\") " pod="opendatahub/test-curl-7753-140460068331952" Jul 16 09:35:30.618996 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.618975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxxbt\" (UniqueName: \"kubernetes.io/projected/f3539ebf-c534-4247-b9b3-3b5ce19b366b-kube-api-access-xxxbt\") pod \"test-curl-7753-140460068331952\" (UID: \"f3539ebf-c534-4247-b9b3-3b5ce19b366b\") " pod="opendatahub/test-curl-7753-140460068331952" Jul 16 09:35:30.627919 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.627900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxxbt\" (UniqueName: \"kubernetes.io/projected/f3539ebf-c534-4247-b9b3-3b5ce19b366b-kube-api-access-xxxbt\") pod \"test-curl-7753-140460068331952\" (UID: \"f3539ebf-c534-4247-b9b3-3b5ce19b366b\") " pod="opendatahub/test-curl-7753-140460068331952" Jul 16 09:35:30.667378 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.667361 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460068331952" Jul 16 09:35:30.790666 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.790634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460068331952"] Jul 16 09:35:30.793979 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:35:30.793953 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3539ebf_c534_4247_b9b3_3b5ce19b366b.slice/crio-b555aca8a4a570cb590b107ca5a2f3308dfc997bb92e4053384cf89e6a1ca407 WatchSource:0}: Error finding container b555aca8a4a570cb590b107ca5a2f3308dfc997bb92e4053384cf89e6a1ca407: Status 404 returned error can't find the container with id b555aca8a4a570cb590b107ca5a2f3308dfc997bb92e4053384cf89e6a1ca407 Jul 16 09:35:30.795717 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:30.795696 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 09:35:31.155470 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:31.155433 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460068331952" event={"ID":"f3539ebf-c534-4247-b9b3-3b5ce19b366b","Type":"ContainerStarted","Data":"b555aca8a4a570cb590b107ca5a2f3308dfc997bb92e4053384cf89e6a1ca407"} Jul 16 09:35:32.161538 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:32.161504 2574 generic.go:356] "Generic (PLEG): container finished" podID="f3539ebf-c534-4247-b9b3-3b5ce19b366b" containerID="1902e7af49ccf5db3e24d902fe09f418aa9c605b3fb57b63ce2cc6bf8eb34a4a" exitCode=0 Jul 16 09:35:32.161887 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:32.161589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460068331952" event={"ID":"f3539ebf-c534-4247-b9b3-3b5ce19b366b","Type":"ContainerDied","Data":"1902e7af49ccf5db3e24d902fe09f418aa9c605b3fb57b63ce2cc6bf8eb34a4a"} Jul 16 09:35:33.295460 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.295434 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460068331952" Jul 16 09:35:33.329574 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.329554 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7753-140460068331952_f3539ebf-c534-4247-b9b3-3b5ce19b366b/test-curl-7753-140460068331952/0.log" Jul 16 09:35:33.339400 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.339383 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxxbt\" (UniqueName: \"kubernetes.io/projected/f3539ebf-c534-4247-b9b3-3b5ce19b366b-kube-api-access-xxxbt\") pod \"f3539ebf-c534-4247-b9b3-3b5ce19b366b\" (UID: \"f3539ebf-c534-4247-b9b3-3b5ce19b366b\") " Jul 16 09:35:33.341394 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.341362 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3539ebf-c534-4247-b9b3-3b5ce19b366b-kube-api-access-xxxbt" (OuterVolumeSpecName: "kube-api-access-xxxbt") pod "f3539ebf-c534-4247-b9b3-3b5ce19b366b" (UID: "f3539ebf-c534-4247-b9b3-3b5ce19b366b"). InnerVolumeSpecName "kube-api-access-xxxbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:35:33.354939 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.354909 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7753-140460068331952"] Jul 16 09:35:33.359041 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.359022 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7753-140460068331952"] Jul 16 09:35:33.440853 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.440802 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxxbt\" (UniqueName: \"kubernetes.io/projected/f3539ebf-c534-4247-b9b3-3b5ce19b366b-kube-api-access-xxxbt\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:33.489704 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.489681 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7753-140460068334640"] Jul 16 09:35:33.490005 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.489991 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3539ebf-c534-4247-b9b3-3b5ce19b366b" containerName="test-curl-7753-140460068331952" Jul 16 09:35:33.490054 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.490006 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3539ebf-c534-4247-b9b3-3b5ce19b366b" containerName="test-curl-7753-140460068331952" Jul 16 09:35:33.490091 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.490062 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3539ebf-c534-4247-b9b3-3b5ce19b366b" containerName="test-curl-7753-140460068331952" Jul 16 09:35:33.492898 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.492884 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460068334640" Jul 16 09:35:33.500144 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.499939 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460068334640"] Jul 16 09:35:33.542183 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.542143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrfg\" (UniqueName: \"kubernetes.io/projected/4cdf3e50-88f0-467b-9c3b-ac2585e2910a-kube-api-access-jkrfg\") pod \"test-curl-7753-140460068334640\" (UID: \"4cdf3e50-88f0-467b-9c3b-ac2585e2910a\") " pod="opendatahub/test-curl-7753-140460068334640" Jul 16 09:35:33.643483 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.643461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrfg\" (UniqueName: \"kubernetes.io/projected/4cdf3e50-88f0-467b-9c3b-ac2585e2910a-kube-api-access-jkrfg\") pod \"test-curl-7753-140460068334640\" (UID: \"4cdf3e50-88f0-467b-9c3b-ac2585e2910a\") " pod="opendatahub/test-curl-7753-140460068334640" Jul 16 09:35:33.652038 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.652011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrfg\" (UniqueName: \"kubernetes.io/projected/4cdf3e50-88f0-467b-9c3b-ac2585e2910a-kube-api-access-jkrfg\") pod \"test-curl-7753-140460068334640\" (UID: \"4cdf3e50-88f0-467b-9c3b-ac2585e2910a\") " pod="opendatahub/test-curl-7753-140460068334640" Jul 16 09:35:33.803326 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.803276 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460068334640" Jul 16 09:35:33.930041 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:33.930016 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460068334640"] Jul 16 09:35:33.932331 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:35:33.932299 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cdf3e50_88f0_467b_9c3b_ac2585e2910a.slice/crio-bfa1b3003f1132d60a7da0c1636c010454af776022817b7004ac23c452b62681 WatchSource:0}: Error finding container bfa1b3003f1132d60a7da0c1636c010454af776022817b7004ac23c452b62681: Status 404 returned error can't find the container with id bfa1b3003f1132d60a7da0c1636c010454af776022817b7004ac23c452b62681 Jul 16 09:35:34.170619 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:34.170586 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460068334640" event={"ID":"4cdf3e50-88f0-467b-9c3b-ac2585e2910a","Type":"ContainerStarted","Data":"bfa1b3003f1132d60a7da0c1636c010454af776022817b7004ac23c452b62681"} Jul 16 09:35:34.171774 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:34.171755 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b555aca8a4a570cb590b107ca5a2f3308dfc997bb92e4053384cf89e6a1ca407" Jul 16 09:35:34.171887 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:34.171773 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460068331952" Jul 16 09:35:34.181797 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:35:34.181769 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7753-140460068331952\" is forbidden: User \"system:node:ip-10-0-128-59.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-59.ec2.internal' and this object" podUID="f3539ebf-c534-4247-b9b3-3b5ce19b366b" pod="opendatahub/test-curl-7753-140460068331952" Jul 16 09:35:35.056599 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:35.056560 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3539ebf-c534-4247-b9b3-3b5ce19b366b" path="/var/lib/kubelet/pods/f3539ebf-c534-4247-b9b3-3b5ce19b366b/volumes" Jul 16 09:35:35.178129 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:35.178049 2574 generic.go:356] "Generic (PLEG): container finished" podID="4cdf3e50-88f0-467b-9c3b-ac2585e2910a" containerID="7e0bd4196bfb0f6167acea3b25ef56cfd7483012227cf4602f0b64385c490c3f" exitCode=0 Jul 16 09:35:35.178272 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:35.178148 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460068334640" event={"ID":"4cdf3e50-88f0-467b-9c3b-ac2585e2910a","Type":"ContainerDied","Data":"7e0bd4196bfb0f6167acea3b25ef56cfd7483012227cf4602f0b64385c490c3f"} Jul 16 09:35:36.320383 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.320361 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460068334640" Jul 16 09:35:36.341205 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.341180 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7753-140460068334640_4cdf3e50-88f0-467b-9c3b-ac2585e2910a/test-curl-7753-140460068334640/0.log" Jul 16 09:35:36.365746 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.365720 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7753-140460068334640"] Jul 16 09:35:36.370350 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.370328 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7753-140460068334640"] Jul 16 09:35:36.465601 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.465543 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkrfg\" (UniqueName: \"kubernetes.io/projected/4cdf3e50-88f0-467b-9c3b-ac2585e2910a-kube-api-access-jkrfg\") pod \"4cdf3e50-88f0-467b-9c3b-ac2585e2910a\" (UID: \"4cdf3e50-88f0-467b-9c3b-ac2585e2910a\") " Jul 16 09:35:36.467452 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.467431 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdf3e50-88f0-467b-9c3b-ac2585e2910a-kube-api-access-jkrfg" (OuterVolumeSpecName: "kube-api-access-jkrfg") pod "4cdf3e50-88f0-467b-9c3b-ac2585e2910a" (UID: "4cdf3e50-88f0-467b-9c3b-ac2585e2910a"). InnerVolumeSpecName "kube-api-access-jkrfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:35:36.542040 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.542012 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:36.542376 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.542362 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdf3e50-88f0-467b-9c3b-ac2585e2910a" containerName="test-curl-7753-140460068334640" Jul 16 09:35:36.542425 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.542377 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdf3e50-88f0-467b-9c3b-ac2585e2910a" containerName="test-curl-7753-140460068334640" Jul 16 09:35:36.542460 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.542443 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdf3e50-88f0-467b-9c3b-ac2585e2910a" containerName="test-curl-7753-140460068334640" Jul 16 09:35:36.545141 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.545125 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:36.552440 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.552419 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:36.566321 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.566298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnbw\" (UniqueName: \"kubernetes.io/projected/96344728-fb37-4620-a275-6befff9d17c6-kube-api-access-6vnbw\") pod \"test-curl-7753-140460064664864\" (UID: \"96344728-fb37-4620-a275-6befff9d17c6\") " pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:36.566438 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.566416 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkrfg\" (UniqueName: \"kubernetes.io/projected/4cdf3e50-88f0-467b-9c3b-ac2585e2910a-kube-api-access-jkrfg\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:36.667531 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.667507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnbw\" (UniqueName: \"kubernetes.io/projected/96344728-fb37-4620-a275-6befff9d17c6-kube-api-access-6vnbw\") pod \"test-curl-7753-140460064664864\" (UID: \"96344728-fb37-4620-a275-6befff9d17c6\") " pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:36.675978 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.675956 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnbw\" (UniqueName: \"kubernetes.io/projected/96344728-fb37-4620-a275-6befff9d17c6-kube-api-access-6vnbw\") pod \"test-curl-7753-140460064664864\" (UID: \"96344728-fb37-4620-a275-6befff9d17c6\") " pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:36.854540 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.854476 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:36.981137 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:36.981109 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:36.985062 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:35:36.985038 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96344728_fb37_4620_a275_6befff9d17c6.slice/crio-f2ae11d8e2ad8c4c1bc7c0c23f020f930454322623fd1586df4fd1531b944380 WatchSource:0}: Error finding container f2ae11d8e2ad8c4c1bc7c0c23f020f930454322623fd1586df4fd1531b944380: Status 404 returned error can't find the container with id f2ae11d8e2ad8c4c1bc7c0c23f020f930454322623fd1586df4fd1531b944380 Jul 16 09:35:37.055483 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:37.055459 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cdf3e50-88f0-467b-9c3b-ac2585e2910a" path="/var/lib/kubelet/pods/4cdf3e50-88f0-467b-9c3b-ac2585e2910a/volumes" Jul 16 09:35:37.188745 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:37.188723 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460068334640" Jul 16 09:35:37.188867 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:37.188765 2574 scope.go:117] "RemoveContainer" containerID="7e0bd4196bfb0f6167acea3b25ef56cfd7483012227cf4602f0b64385c490c3f" Jul 16 09:35:37.190179 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:37.190051 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460064664864" event={"ID":"96344728-fb37-4620-a275-6befff9d17c6","Type":"ContainerStarted","Data":"f2ae11d8e2ad8c4c1bc7c0c23f020f930454322623fd1586df4fd1531b944380"} Jul 16 09:35:38.197295 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:38.197256 2574 generic.go:356] "Generic (PLEG): container finished" podID="96344728-fb37-4620-a275-6befff9d17c6" containerID="03899c95115ffee4b631115e32e7a9fdb547a57edcfe189f8816be839420a23b" exitCode=0 Jul 16 09:35:38.197657 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:38.197330 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460064664864" event={"ID":"96344728-fb37-4620-a275-6befff9d17c6","Type":"ContainerDied","Data":"03899c95115ffee4b631115e32e7a9fdb547a57edcfe189f8816be839420a23b"} Jul 16 09:35:39.335136 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.335111 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:39.356425 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.356395 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7753-140460064664864_96344728-fb37-4620-a275-6befff9d17c6/test-curl-7753-140460064664864/0.log" Jul 16 09:35:39.383124 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.383102 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:39.385246 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.385225 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:39.390865 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.390844 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vnbw\" (UniqueName: \"kubernetes.io/projected/96344728-fb37-4620-a275-6befff9d17c6-kube-api-access-6vnbw\") pod \"96344728-fb37-4620-a275-6befff9d17c6\" (UID: \"96344728-fb37-4620-a275-6befff9d17c6\") " Jul 16 09:35:39.392654 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.392631 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96344728-fb37-4620-a275-6befff9d17c6-kube-api-access-6vnbw" (OuterVolumeSpecName: "kube-api-access-6vnbw") pod "96344728-fb37-4620-a275-6befff9d17c6" (UID: "96344728-fb37-4620-a275-6befff9d17c6"). InnerVolumeSpecName "kube-api-access-6vnbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:35:39.492302 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.492242 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vnbw\" (UniqueName: \"kubernetes.io/projected/96344728-fb37-4620-a275-6befff9d17c6-kube-api-access-6vnbw\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:39.518223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.515661 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:39.518223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.516394 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96344728-fb37-4620-a275-6befff9d17c6" containerName="test-curl-7753-140460064664864" Jul 16 09:35:39.518223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.516414 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="96344728-fb37-4620-a275-6befff9d17c6" containerName="test-curl-7753-140460064664864" Jul 16 09:35:39.518223 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.516593 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="96344728-fb37-4620-a275-6befff9d17c6" containerName="test-curl-7753-140460064664864" Jul 16 09:35:39.520408 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.520389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:39.523255 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.523235 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:39.593388 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.593354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbhxf\" (UniqueName: \"kubernetes.io/projected/4b51f38d-440a-4522-9ba9-d162d38d6e6f-kube-api-access-mbhxf\") pod \"test-curl-7753-140460064664864\" (UID: \"4b51f38d-440a-4522-9ba9-d162d38d6e6f\") " pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:39.694599 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.694572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbhxf\" (UniqueName: \"kubernetes.io/projected/4b51f38d-440a-4522-9ba9-d162d38d6e6f-kube-api-access-mbhxf\") pod \"test-curl-7753-140460064664864\" (UID: \"4b51f38d-440a-4522-9ba9-d162d38d6e6f\") " pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:39.703905 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.703881 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbhxf\" (UniqueName: \"kubernetes.io/projected/4b51f38d-440a-4522-9ba9-d162d38d6e6f-kube-api-access-mbhxf\") pod \"test-curl-7753-140460064664864\" (UID: \"4b51f38d-440a-4522-9ba9-d162d38d6e6f\") " pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:39.830038 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.829981 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:39.951769 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:39.951744 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:39.953812 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:35:39.953786 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b51f38d_440a_4522_9ba9_d162d38d6e6f.slice/crio-ec050774bd553749ee49d0a85c92e9c46394e8761d6907f3c447960f20a2fb13 WatchSource:0}: Error finding container ec050774bd553749ee49d0a85c92e9c46394e8761d6907f3c447960f20a2fb13: Status 404 returned error can't find the container with id ec050774bd553749ee49d0a85c92e9c46394e8761d6907f3c447960f20a2fb13 Jul 16 09:35:40.209931 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:40.209904 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:40.210043 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:40.209928 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ae11d8e2ad8c4c1bc7c0c23f020f930454322623fd1586df4fd1531b944380" Jul 16 09:35:40.211065 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:40.211044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460064664864" event={"ID":"4b51f38d-440a-4522-9ba9-d162d38d6e6f","Type":"ContainerStarted","Data":"ec050774bd553749ee49d0a85c92e9c46394e8761d6907f3c447960f20a2fb13"} Jul 16 09:35:40.221824 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:40.221795 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7753-140460064664864" oldPodUID="96344728-fb37-4620-a275-6befff9d17c6" podUID="4b51f38d-440a-4522-9ba9-d162d38d6e6f" Jul 16 09:35:40.420493 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:40.420464 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:35:41.056500 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:41.056465 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96344728-fb37-4620-a275-6befff9d17c6" path="/var/lib/kubelet/pods/96344728-fb37-4620-a275-6befff9d17c6/volumes" Jul 16 09:35:41.216992 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:41.216964 2574 generic.go:356] "Generic (PLEG): container finished" podID="4b51f38d-440a-4522-9ba9-d162d38d6e6f" containerID="de77e33a01990024978a40965dcd670ff6fdca3e5ae9ea6a867aee7fbb88e35e" exitCode=0 Jul 16 09:35:41.217104 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:41.217055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460064664864" event={"ID":"4b51f38d-440a-4522-9ba9-d162d38d6e6f","Type":"ContainerDied","Data":"de77e33a01990024978a40965dcd670ff6fdca3e5ae9ea6a867aee7fbb88e35e"} Jul 16 09:35:42.358031 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.358009 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:42.378992 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.378970 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7753-140460064664864_4b51f38d-440a-4522-9ba9-d162d38d6e6f/test-curl-7753-140460064664864/0.log" Jul 16 09:35:42.405256 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.405229 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:42.407476 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.407454 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7753-140460064664864"] Jul 16 09:35:42.414266 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.414249 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbhxf\" (UniqueName: \"kubernetes.io/projected/4b51f38d-440a-4522-9ba9-d162d38d6e6f-kube-api-access-mbhxf\") pod \"4b51f38d-440a-4522-9ba9-d162d38d6e6f\" (UID: \"4b51f38d-440a-4522-9ba9-d162d38d6e6f\") " Jul 16 09:35:42.416095 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.416074 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b51f38d-440a-4522-9ba9-d162d38d6e6f-kube-api-access-mbhxf" (OuterVolumeSpecName: "kube-api-access-mbhxf") pod "4b51f38d-440a-4522-9ba9-d162d38d6e6f" (UID: "4b51f38d-440a-4522-9ba9-d162d38d6e6f"). InnerVolumeSpecName "kube-api-access-mbhxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:35:42.514938 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.514878 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbhxf\" (UniqueName: \"kubernetes.io/projected/4b51f38d-440a-4522-9ba9-d162d38d6e6f-kube-api-access-mbhxf\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:42.541037 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.541013 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:42.541375 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.541361 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b51f38d-440a-4522-9ba9-d162d38d6e6f" containerName="test-curl-7753-140460064664864" Jul 16 09:35:42.541417 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.541376 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b51f38d-440a-4522-9ba9-d162d38d6e6f" containerName="test-curl-7753-140460064664864" Jul 16 09:35:42.541448 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.541443 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b51f38d-440a-4522-9ba9-d162d38d6e6f" containerName="test-curl-7753-140460064664864" Jul 16 09:35:42.544344 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.544329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:42.550961 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.550937 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:42.615713 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.615692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxrt\" (UniqueName: \"kubernetes.io/projected/f82a189d-7501-463c-a18e-c984965213e9-kube-api-access-dsxrt\") pod \"test-curl-7753-140460067885392\" (UID: \"f82a189d-7501-463c-a18e-c984965213e9\") " pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:42.717010 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.716985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxrt\" (UniqueName: \"kubernetes.io/projected/f82a189d-7501-463c-a18e-c984965213e9-kube-api-access-dsxrt\") pod \"test-curl-7753-140460067885392\" (UID: \"f82a189d-7501-463c-a18e-c984965213e9\") " pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:42.726124 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.726082 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxrt\" (UniqueName: \"kubernetes.io/projected/f82a189d-7501-463c-a18e-c984965213e9-kube-api-access-dsxrt\") pod \"test-curl-7753-140460067885392\" (UID: \"f82a189d-7501-463c-a18e-c984965213e9\") " pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:42.854034 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:42.853974 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:43.007481 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.007456 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:43.009736 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:35:43.009707 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf82a189d_7501_463c_a18e_c984965213e9.slice/crio-ac7cc93c299e5526b9485e3127fc7bc5f41ae836ff77fa4d8b9d625253d656dd WatchSource:0}: Error finding container ac7cc93c299e5526b9485e3127fc7bc5f41ae836ff77fa4d8b9d625253d656dd: Status 404 returned error can't find the container with id ac7cc93c299e5526b9485e3127fc7bc5f41ae836ff77fa4d8b9d625253d656dd Jul 16 09:35:43.055277 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.055254 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b51f38d-440a-4522-9ba9-d162d38d6e6f" path="/var/lib/kubelet/pods/4b51f38d-440a-4522-9ba9-d162d38d6e6f/volumes" Jul 16 09:35:43.228633 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.228605 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460067885392" event={"ID":"f82a189d-7501-463c-a18e-c984965213e9","Type":"ContainerStarted","Data":"ac7cc93c299e5526b9485e3127fc7bc5f41ae836ff77fa4d8b9d625253d656dd"} Jul 16 09:35:43.229741 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.229725 2574 scope.go:117] "RemoveContainer" containerID="de77e33a01990024978a40965dcd670ff6fdca3e5ae9ea6a867aee7fbb88e35e" Jul 16 09:35:43.229821 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.229742 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460064664864" Jul 16 09:35:43.353338 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.353305 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" podUID="671090cf-5723-418e-b055-b0c8c5c81eb9" containerName="main" containerID="cri-o://3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb" gracePeriod=45 Jul 16 09:35:43.598578 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.598556 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:35:43.623858 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.623834 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-model-cache\") pod \"671090cf-5723-418e-b055-b0c8c5c81eb9\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " Jul 16 09:35:43.623961 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.623908 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swwdq\" (UniqueName: \"kubernetes.io/projected/671090cf-5723-418e-b055-b0c8c5c81eb9-kube-api-access-swwdq\") pod \"671090cf-5723-418e-b055-b0c8c5c81eb9\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " Jul 16 09:35:43.624017 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.623978 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-dshm\") pod \"671090cf-5723-418e-b055-b0c8c5c81eb9\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " Jul 16 09:35:43.624017 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.624002 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671090cf-5723-418e-b055-b0c8c5c81eb9-tls-certs\") pod \"671090cf-5723-418e-b055-b0c8c5c81eb9\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " Jul 16 09:35:43.624108 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.624078 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-tmp-dir\") pod \"671090cf-5723-418e-b055-b0c8c5c81eb9\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " Jul 16 09:35:43.624108 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.624084 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "671090cf-5723-418e-b055-b0c8c5c81eb9" (UID: "671090cf-5723-418e-b055-b0c8c5c81eb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:35:43.624108 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.624100 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-home\") pod \"671090cf-5723-418e-b055-b0c8c5c81eb9\" (UID: \"671090cf-5723-418e-b055-b0c8c5c81eb9\") " Jul 16 09:35:43.624396 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.624374 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-model-cache\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:43.624536 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.624375 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "671090cf-5723-418e-b055-b0c8c5c81eb9" (UID: "671090cf-5723-418e-b055-b0c8c5c81eb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:35:43.624536 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.624452 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-home" (OuterVolumeSpecName: "home") pod "671090cf-5723-418e-b055-b0c8c5c81eb9" (UID: "671090cf-5723-418e-b055-b0c8c5c81eb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:35:43.626286 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.626223 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/671090cf-5723-418e-b055-b0c8c5c81eb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "671090cf-5723-418e-b055-b0c8c5c81eb9" (UID: "671090cf-5723-418e-b055-b0c8c5c81eb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 09:35:43.626512 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.626494 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-dshm" (OuterVolumeSpecName: "dshm") pod "671090cf-5723-418e-b055-b0c8c5c81eb9" (UID: "671090cf-5723-418e-b055-b0c8c5c81eb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:35:43.626556 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.626520 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/671090cf-5723-418e-b055-b0c8c5c81eb9-kube-api-access-swwdq" (OuterVolumeSpecName: "kube-api-access-swwdq") pod "671090cf-5723-418e-b055-b0c8c5c81eb9" (UID: "671090cf-5723-418e-b055-b0c8c5c81eb9"). InnerVolumeSpecName "kube-api-access-swwdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:35:43.725018 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.724992 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-dshm\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:43.725107 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.725029 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/671090cf-5723-418e-b055-b0c8c5c81eb9-tls-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:43.725107 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.725039 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-tmp-dir\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:43.725107 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.725047 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/671090cf-5723-418e-b055-b0c8c5c81eb9-home\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:43.725107 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:43.725056 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swwdq\" (UniqueName: \"kubernetes.io/projected/671090cf-5723-418e-b055-b0c8c5c81eb9-kube-api-access-swwdq\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:44.237302 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.237269 2574 generic.go:356] "Generic (PLEG): container finished" podID="f82a189d-7501-463c-a18e-c984965213e9" containerID="d737e328e5221b8191d6c6b0d821f7e06efbf5caa2852f97be6bae55d33eaca7" exitCode=0 Jul 16 09:35:44.237451 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.237347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460067885392" event={"ID":"f82a189d-7501-463c-a18e-c984965213e9","Type":"ContainerDied","Data":"d737e328e5221b8191d6c6b0d821f7e06efbf5caa2852f97be6bae55d33eaca7"} Jul 16 09:35:44.239541 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.239517 2574 generic.go:356] "Generic (PLEG): container finished" podID="671090cf-5723-418e-b055-b0c8c5c81eb9" containerID="3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb" exitCode=0 Jul 16 09:35:44.239648 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.239572 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" Jul 16 09:35:44.239720 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.239575 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" event={"ID":"671090cf-5723-418e-b055-b0c8c5c81eb9","Type":"ContainerDied","Data":"3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb"} Jul 16 09:35:44.239720 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.239692 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv" event={"ID":"671090cf-5723-418e-b055-b0c8c5c81eb9","Type":"ContainerDied","Data":"e4e4a16edb598ca783e50fcb91d9b4e2e54a8508edd4c4affafa8d5d0703ddd4"} Jul 16 09:35:44.239827 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.239723 2574 scope.go:117] "RemoveContainer" containerID="3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb" Jul 16 09:35:44.248608 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.248591 2574 scope.go:117] "RemoveContainer" containerID="3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb" Jul 16 09:35:44.248885 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:35:44.248866 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb\": container with ID starting with 3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb not found: ID does not exist" containerID="3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb" Jul 16 09:35:44.248948 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.248894 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb"} err="failed to get container status \"3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb\": rpc error: code = NotFound desc = could not find container \"3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb\": container with ID starting with 3e6036992b50e210420c639a85dc7d4a8964997427f49b5bb206928f88ee10fb not found: ID does not exist" Jul 16 09:35:44.266042 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.266016 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv"] Jul 16 09:35:44.271168 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:44.271127 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-923c3a/rate-test-model-923c3a-kserve-75864884f5-872jv"] Jul 16 09:35:45.056469 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.056436 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="671090cf-5723-418e-b055-b0c8c5c81eb9" path="/var/lib/kubelet/pods/671090cf-5723-418e-b055-b0c8c5c81eb9/volumes" Jul 16 09:35:45.380257 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.380199 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:45.401955 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.401926 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7753-140460067885392_f82a189d-7501-463c-a18e-c984965213e9/test-curl-7753-140460067885392/0.log" Jul 16 09:35:45.430637 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.430605 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:45.434496 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.434474 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:45.438200 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.438179 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsxrt\" (UniqueName: \"kubernetes.io/projected/f82a189d-7501-463c-a18e-c984965213e9-kube-api-access-dsxrt\") pod \"f82a189d-7501-463c-a18e-c984965213e9\" (UID: \"f82a189d-7501-463c-a18e-c984965213e9\") " Jul 16 09:35:45.440012 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.439987 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82a189d-7501-463c-a18e-c984965213e9-kube-api-access-dsxrt" (OuterVolumeSpecName: "kube-api-access-dsxrt") pod "f82a189d-7501-463c-a18e-c984965213e9" (UID: "f82a189d-7501-463c-a18e-c984965213e9"). InnerVolumeSpecName "kube-api-access-dsxrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:35:45.539603 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.539580 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsxrt\" (UniqueName: \"kubernetes.io/projected/f82a189d-7501-463c-a18e-c984965213e9-kube-api-access-dsxrt\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:45.568446 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.568424 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:45.568757 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.568745 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="671090cf-5723-418e-b055-b0c8c5c81eb9" containerName="main" Jul 16 09:35:45.568813 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.568759 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="671090cf-5723-418e-b055-b0c8c5c81eb9" containerName="main" Jul 16 09:35:45.568813 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.568773 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f82a189d-7501-463c-a18e-c984965213e9" containerName="test-curl-7753-140460067885392" Jul 16 09:35:45.568813 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.568778 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f82a189d-7501-463c-a18e-c984965213e9" containerName="test-curl-7753-140460067885392" Jul 16 09:35:45.568923 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.568828 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="671090cf-5723-418e-b055-b0c8c5c81eb9" containerName="main" Jul 16 09:35:45.568923 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.568836 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f82a189d-7501-463c-a18e-c984965213e9" containerName="test-curl-7753-140460067885392" Jul 16 09:35:45.571556 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.571541 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:45.578099 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.578057 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:45.640409 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.640389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g84br\" (UniqueName: \"kubernetes.io/projected/6174c1f2-9670-428e-a777-bacc427c11c5-kube-api-access-g84br\") pod \"test-curl-7753-140460067885392\" (UID: \"6174c1f2-9670-428e-a777-bacc427c11c5\") " pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:45.741678 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.741656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g84br\" (UniqueName: \"kubernetes.io/projected/6174c1f2-9670-428e-a777-bacc427c11c5-kube-api-access-g84br\") pod \"test-curl-7753-140460067885392\" (UID: \"6174c1f2-9670-428e-a777-bacc427c11c5\") " pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:45.750966 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.750945 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g84br\" (UniqueName: \"kubernetes.io/projected/6174c1f2-9670-428e-a777-bacc427c11c5-kube-api-access-g84br\") pod \"test-curl-7753-140460067885392\" (UID: \"6174c1f2-9670-428e-a777-bacc427c11c5\") " pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:45.880712 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:45.880692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:46.007376 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:46.007348 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:46.008519 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:35:46.008494 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6174c1f2_9670_428e_a777_bacc427c11c5.slice/crio-317b4eb02bf837e158fe45597b75dd971bed83180cff16e986b76984c3980864 WatchSource:0}: Error finding container 317b4eb02bf837e158fe45597b75dd971bed83180cff16e986b76984c3980864: Status 404 returned error can't find the container with id 317b4eb02bf837e158fe45597b75dd971bed83180cff16e986b76984c3980864 Jul 16 09:35:46.251106 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:46.251039 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:46.251106 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:46.251088 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac7cc93c299e5526b9485e3127fc7bc5f41ae836ff77fa4d8b9d625253d656dd" Jul 16 09:35:46.252376 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:46.252350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460067885392" event={"ID":"6174c1f2-9670-428e-a777-bacc427c11c5","Type":"ContainerStarted","Data":"317b4eb02bf837e158fe45597b75dd971bed83180cff16e986b76984c3980864"} Jul 16 09:35:46.263763 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:46.263741 2574 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7753-140460067885392" oldPodUID="f82a189d-7501-463c-a18e-c984965213e9" podUID="6174c1f2-9670-428e-a777-bacc427c11c5" Jul 16 09:35:47.056725 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:47.056693 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f82a189d-7501-463c-a18e-c984965213e9" path="/var/lib/kubelet/pods/f82a189d-7501-463c-a18e-c984965213e9/volumes" Jul 16 09:35:47.258673 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:47.258650 2574 generic.go:356] "Generic (PLEG): container finished" podID="6174c1f2-9670-428e-a777-bacc427c11c5" containerID="a518d511342d61caf8495eda331603d3f7ce17d5c3f64c57bc938e97ed667b7e" exitCode=0 Jul 16 09:35:47.258959 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:47.258688 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7753-140460067885392" event={"ID":"6174c1f2-9670-428e-a777-bacc427c11c5","Type":"ContainerDied","Data":"a518d511342d61caf8495eda331603d3f7ce17d5c3f64c57bc938e97ed667b7e"} Jul 16 09:35:48.396890 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:48.396864 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:48.422577 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:48.422553 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7753-140460067885392_6174c1f2-9670-428e-a777-bacc427c11c5/test-curl-7753-140460067885392/0.log" Jul 16 09:35:48.448774 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:48.448749 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:48.453662 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:48.453641 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7753-140460067885392"] Jul 16 09:35:48.463486 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:48.463463 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g84br\" (UniqueName: \"kubernetes.io/projected/6174c1f2-9670-428e-a777-bacc427c11c5-kube-api-access-g84br\") pod \"6174c1f2-9670-428e-a777-bacc427c11c5\" (UID: \"6174c1f2-9670-428e-a777-bacc427c11c5\") " Jul 16 09:35:48.465364 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:48.465342 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6174c1f2-9670-428e-a777-bacc427c11c5-kube-api-access-g84br" (OuterVolumeSpecName: "kube-api-access-g84br") pod "6174c1f2-9670-428e-a777-bacc427c11c5" (UID: "6174c1f2-9670-428e-a777-bacc427c11c5"). InnerVolumeSpecName "kube-api-access-g84br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:35:48.565063 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:48.565004 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g84br\" (UniqueName: \"kubernetes.io/projected/6174c1f2-9670-428e-a777-bacc427c11c5-kube-api-access-g84br\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:35:49.055559 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:49.055532 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6174c1f2-9670-428e-a777-bacc427c11c5" path="/var/lib/kubelet/pods/6174c1f2-9670-428e-a777-bacc427c11c5/volumes" Jul 16 09:35:49.269317 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:49.269293 2574 scope.go:117] "RemoveContainer" containerID="a518d511342d61caf8495eda331603d3f7ce17d5c3f64c57bc938e97ed667b7e" Jul 16 09:35:49.269317 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:49.269305 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7753-140460067885392" Jul 16 09:35:59.019441 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:35:59.019399 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:36:01.272062 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:01.272020 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj"] Jul 16 09:36:01.272578 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:01.272288 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" containerID="cri-o://7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3" gracePeriod=30 Jul 16 09:36:06.517089 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.517061 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:36:06.621252 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621226 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-data\") pod \"524a0c18-f938-4c76-8833-ce62a32465ea\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " Jul 16 09:36:06.621423 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621258 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/524a0c18-f938-4c76-8833-ce62a32465ea-istiod-ca-cert\") pod \"524a0c18-f938-4c76-8833-ce62a32465ea\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " Jul 16 09:36:06.621423 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621305 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-envoy\") pod \"524a0c18-f938-4c76-8833-ce62a32465ea\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " Jul 16 09:36:06.621423 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621328 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d52fw\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-kube-api-access-d52fw\") pod \"524a0c18-f938-4c76-8833-ce62a32465ea\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " Jul 16 09:36:06.621599 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621479 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-certs\") pod \"524a0c18-f938-4c76-8833-ce62a32465ea\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " Jul 16 09:36:06.621599 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621565 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/524a0c18-f938-4c76-8833-ce62a32465ea-istio-podinfo\") pod \"524a0c18-f938-4c76-8833-ce62a32465ea\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " Jul 16 09:36:06.621698 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621614 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-istio-token\") pod \"524a0c18-f938-4c76-8833-ce62a32465ea\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " Jul 16 09:36:06.621781 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621672 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/524a0c18-f938-4c76-8833-ce62a32465ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "524a0c18-f938-4c76-8833-ce62a32465ea" (UID: "524a0c18-f938-4c76-8833-ce62a32465ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 09:36:06.621866 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621759 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "524a0c18-f938-4c76-8833-ce62a32465ea" (UID: "524a0c18-f938-4c76-8833-ce62a32465ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:06.621866 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621829 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-socket\") pod \"524a0c18-f938-4c76-8833-ce62a32465ea\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " Jul 16 09:36:06.621969 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.621880 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-credential-socket\") pod \"524a0c18-f938-4c76-8833-ce62a32465ea\" (UID: \"524a0c18-f938-4c76-8833-ce62a32465ea\") " Jul 16 09:36:06.622133 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.622104 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "524a0c18-f938-4c76-8833-ce62a32465ea" (UID: "524a0c18-f938-4c76-8833-ce62a32465ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:06.622277 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.622258 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:06.622363 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.622281 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-workload-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:06.622363 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.622295 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/524a0c18-f938-4c76-8833-ce62a32465ea-istiod-ca-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:06.622363 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.622270 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "524a0c18-f938-4c76-8833-ce62a32465ea" (UID: "524a0c18-f938-4c76-8833-ce62a32465ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:06.622480 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.622380 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "524a0c18-f938-4c76-8833-ce62a32465ea" (UID: "524a0c18-f938-4c76-8833-ce62a32465ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:06.623714 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.623693 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "524a0c18-f938-4c76-8833-ce62a32465ea" (UID: "524a0c18-f938-4c76-8833-ce62a32465ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:36:06.623795 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.623728 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/524a0c18-f938-4c76-8833-ce62a32465ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "524a0c18-f938-4c76-8833-ce62a32465ea" (UID: "524a0c18-f938-4c76-8833-ce62a32465ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 09:36:06.623947 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.623931 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-kube-api-access-d52fw" (OuterVolumeSpecName: "kube-api-access-d52fw") pod "524a0c18-f938-4c76-8833-ce62a32465ea" (UID: "524a0c18-f938-4c76-8833-ce62a32465ea"). InnerVolumeSpecName "kube-api-access-d52fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:36:06.624003 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.623936 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "524a0c18-f938-4c76-8833-ce62a32465ea" (UID: "524a0c18-f938-4c76-8833-ce62a32465ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:06.723516 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.723485 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/524a0c18-f938-4c76-8833-ce62a32465ea-istio-podinfo\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:06.723516 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.723510 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-istio-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:06.723516 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.723519 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-credential-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:06.723713 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.723529 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-data\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:06.723713 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.723537 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/524a0c18-f938-4c76-8833-ce62a32465ea-istio-envoy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:06.723713 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:06.723545 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d52fw\" (UniqueName: \"kubernetes.io/projected/524a0c18-f938-4c76-8833-ce62a32465ea-kube-api-access-d52fw\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:07.359239 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.359125 2574 generic.go:356] "Generic (PLEG): container finished" podID="524a0c18-f938-4c76-8833-ce62a32465ea" containerID="7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3" exitCode=0 Jul 16 09:36:07.359239 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.359195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" event={"ID":"524a0c18-f938-4c76-8833-ce62a32465ea","Type":"ContainerDied","Data":"7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3"} Jul 16 09:36:07.359239 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.359222 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" Jul 16 09:36:07.359549 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.359286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj" event={"ID":"524a0c18-f938-4c76-8833-ce62a32465ea","Type":"ContainerDied","Data":"05ee81893a2d5a398fb2aabac030017aab5602fc3323a3963c1d43c65fa79f4c"} Jul 16 09:36:07.359549 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.359319 2574 scope.go:117] "RemoveContainer" containerID="7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3" Jul 16 09:36:07.368011 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.367991 2574 scope.go:117] "RemoveContainer" containerID="872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709" Jul 16 09:36:07.375822 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.375695 2574 scope.go:117] "RemoveContainer" containerID="7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3" Jul 16 09:36:07.376230 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:36:07.376202 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3\": container with ID starting with 7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3 not found: ID does not exist" containerID="7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3" Jul 16 09:36:07.376307 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.376240 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3"} err="failed to get container status \"7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3\": rpc error: code = NotFound desc = could not find container \"7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3\": container with ID starting with 7da351ca5f382dd7bb9291df712e0448930195e33a46b6c7b5f4909cbb4598c3 not found: ID does not exist" Jul 16 09:36:07.376307 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.376264 2574 scope.go:117] "RemoveContainer" containerID="872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709" Jul 16 09:36:07.376547 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:36:07.376524 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709\": container with ID starting with 872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709 not found: ID does not exist" containerID="872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709" Jul 16 09:36:07.376603 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.376552 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709"} err="failed to get container status \"872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709\": rpc error: code = NotFound desc = could not find container \"872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709\": container with ID starting with 872bcfe4d7b501a4d38e535f91af1389b15f286a1d3ba0a5d67a7f853b89d709 not found: ID does not exist" Jul 16 09:36:07.377491 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.377472 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj"] Jul 16 09:36:07.380083 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:07.380065 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-923c3a-openshift-default-5477845cdc-pr9fj"] Jul 16 09:36:09.055645 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:09.055609 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" path="/var/lib/kubelet/pods/524a0c18-f938-4c76-8833-ce62a32465ea/volumes" Jul 16 09:36:10.907834 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:10.907798 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-jl6gr"] Jul 16 09:36:14.370202 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:14.370145 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2"] Jul 16 09:36:14.370651 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:14.370399 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" containerID="cri-o://a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c" gracePeriod=30 Jul 16 09:36:19.059012 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.058967 2574 patch_prober.go:28] interesting pod/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 09:36:19.059506 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.059046 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 09:36:19.632445 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.632393 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:36:19.730334 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730306 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631fb0cc-2136-4bb6-8f63-066d7584d52d-istiod-ca-cert\") pod \"631fb0cc-2136-4bb6-8f63-066d7584d52d\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " Jul 16 09:36:19.730436 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730339 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-socket\") pod \"631fb0cc-2136-4bb6-8f63-066d7584d52d\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " Jul 16 09:36:19.730436 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730356 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-certs\") pod \"631fb0cc-2136-4bb6-8f63-066d7584d52d\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " Jul 16 09:36:19.730436 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730381 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-token\") pod \"631fb0cc-2136-4bb6-8f63-066d7584d52d\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " Jul 16 09:36:19.730436 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730419 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhbdx\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-kube-api-access-mhbdx\") pod \"631fb0cc-2136-4bb6-8f63-066d7584d52d\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " Jul 16 09:36:19.730655 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730451 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-credential-socket\") pod \"631fb0cc-2136-4bb6-8f63-066d7584d52d\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " Jul 16 09:36:19.730655 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730488 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-envoy\") pod \"631fb0cc-2136-4bb6-8f63-066d7584d52d\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " Jul 16 09:36:19.730655 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-podinfo\") pod \"631fb0cc-2136-4bb6-8f63-066d7584d52d\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " Jul 16 09:36:19.730655 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730591 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-data\") pod \"631fb0cc-2136-4bb6-8f63-066d7584d52d\" (UID: \"631fb0cc-2136-4bb6-8f63-066d7584d52d\") " Jul 16 09:36:19.730655 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730647 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "631fb0cc-2136-4bb6-8f63-066d7584d52d" (UID: "631fb0cc-2136-4bb6-8f63-066d7584d52d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:19.730909 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730663 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "631fb0cc-2136-4bb6-8f63-066d7584d52d" (UID: "631fb0cc-2136-4bb6-8f63-066d7584d52d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:19.730909 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730689 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631fb0cc-2136-4bb6-8f63-066d7584d52d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "631fb0cc-2136-4bb6-8f63-066d7584d52d" (UID: "631fb0cc-2136-4bb6-8f63-066d7584d52d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 09:36:19.731023 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730969 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631fb0cc-2136-4bb6-8f63-066d7584d52d-istiod-ca-cert\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:19.731023 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.730988 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:19.731023 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.731002 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-workload-certs\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:19.731300 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.731257 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "631fb0cc-2136-4bb6-8f63-066d7584d52d" (UID: "631fb0cc-2136-4bb6-8f63-066d7584d52d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:19.731648 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.731623 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-data" (OuterVolumeSpecName: "istio-data") pod "631fb0cc-2136-4bb6-8f63-066d7584d52d" (UID: "631fb0cc-2136-4bb6-8f63-066d7584d52d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:19.732712 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.732674 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-token" (OuterVolumeSpecName: "istio-token") pod "631fb0cc-2136-4bb6-8f63-066d7584d52d" (UID: "631fb0cc-2136-4bb6-8f63-066d7584d52d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:36:19.732917 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.732896 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-kube-api-access-mhbdx" (OuterVolumeSpecName: "kube-api-access-mhbdx") pod "631fb0cc-2136-4bb6-8f63-066d7584d52d" (UID: "631fb0cc-2136-4bb6-8f63-066d7584d52d"). InnerVolumeSpecName "kube-api-access-mhbdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 09:36:19.733098 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.733074 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "631fb0cc-2136-4bb6-8f63-066d7584d52d" (UID: "631fb0cc-2136-4bb6-8f63-066d7584d52d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 09:36:19.733098 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.733080 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "631fb0cc-2136-4bb6-8f63-066d7584d52d" (UID: "631fb0cc-2136-4bb6-8f63-066d7584d52d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 09:36:19.832128 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.832102 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbdx\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-kube-api-access-mhbdx\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:19.832128 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.832125 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-credential-socket\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:19.832289 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.832139 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-envoy\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:19.832289 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.832177 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-podinfo\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:19.832289 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.832191 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-data\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:19.832289 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:19.832202 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631fb0cc-2136-4bb6-8f63-066d7584d52d-istio-token\") on node \"ip-10-0-128-59.ec2.internal\" DevicePath \"\"" Jul 16 09:36:20.417930 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.417897 2574 generic.go:356] "Generic (PLEG): container finished" podID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerID="a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c" exitCode=0 Jul 16 09:36:20.418397 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.417944 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" Jul 16 09:36:20.418397 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.417943 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" event={"ID":"631fb0cc-2136-4bb6-8f63-066d7584d52d","Type":"ContainerDied","Data":"a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c"} Jul 16 09:36:20.418397 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.418016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2" event={"ID":"631fb0cc-2136-4bb6-8f63-066d7584d52d","Type":"ContainerDied","Data":"5c98e368e70797e8d3db09919b392aba5c2c1d45ed9737a519e0339ea0359ac1"} Jul 16 09:36:20.418397 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.418051 2574 scope.go:117] "RemoveContainer" containerID="a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c" Jul 16 09:36:20.429679 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.429662 2574 scope.go:117] "RemoveContainer" containerID="74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0" Jul 16 09:36:20.437830 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.437808 2574 scope.go:117] "RemoveContainer" containerID="a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c" Jul 16 09:36:20.438062 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:36:20.438039 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c\": container with ID starting with a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c not found: ID does not exist" containerID="a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c" Jul 16 09:36:20.438141 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.438073 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c"} err="failed to get container status \"a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c\": rpc error: code = NotFound desc = could not find container \"a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c\": container with ID starting with a4e8ff64e066a3c66e85f76151d734eef78a2b65afe9fc4bf6e138beff9f066c not found: ID does not exist" Jul 16 09:36:20.438141 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.438092 2574 scope.go:117] "RemoveContainer" containerID="74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0" Jul 16 09:36:20.438326 ip-10-0-128-59 kubenswrapper[2574]: E0716 09:36:20.438310 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0\": container with ID starting with 74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0 not found: ID does not exist" containerID="74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0" Jul 16 09:36:20.438387 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.438330 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0"} err="failed to get container status \"74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0\": rpc error: code = NotFound desc = could not find container \"74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0\": container with ID starting with 74d871d26180078e856f3c83cdd6d603c8d4e9f6cec7529932fa013ec06fd7e0 not found: ID does not exist" Jul 16 09:36:20.443967 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.443943 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2"] Jul 16 09:36:20.449844 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:20.449823 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-800be8-openshift-default-55ff497447-r6gg2"] Jul 16 09:36:21.055557 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:21.055524 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" path="/var/lib/kubelet/pods/631fb0cc-2136-4bb6-8f63-066d7584d52d/volumes" Jul 16 09:36:21.670614 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:21.670586 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b875787f8-78b79_d8b8fd9d-aebe-4607-b178-df74a27c1d23/copy-manifests/0.log" Jul 16 09:36:21.678310 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:21.678289 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b875787f8-78b79_d8b8fd9d-aebe-4607-b178-df74a27c1d23/manager/0.log" Jul 16 09:36:24.781637 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:24.781598 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jl6gr_73e74bc7-8621-4741-ae0e-e325fe735ec4/limitador/0.log" Jul 16 09:36:25.327386 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:25.327349 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4f9phv_977280d7-4a3a-4fcb-87c4-c863fb79626a/istio-proxy/0.log" Jul 16 09:36:25.576998 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:25.576965 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-856d68f954-94qtz_87d37615-95c8-4a58-86bd-6c7ad1410b29/kube-auth-proxy/0.log" Jul 16 09:36:27.221144 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:27.221115 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-bjq5j_83a87d6d-fc5f-4128-bdd6-1f7823c38f1f/main/0.log" Jul 16 09:36:33.061498 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.061458 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xjqh/must-gather-qj86d"] Jul 16 09:36:33.061890 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.061878 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" Jul 16 09:36:33.061935 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.061894 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" Jul 16 09:36:33.061935 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.061913 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" Jul 16 09:36:33.061935 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.061921 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" Jul 16 09:36:33.062035 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.061935 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" Jul 16 09:36:33.062035 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.061944 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" Jul 16 09:36:33.062035 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.061970 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6174c1f2-9670-428e-a777-bacc427c11c5" containerName="test-curl-7753-140460067885392" Jul 16 09:36:33.062035 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.061979 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6174c1f2-9670-428e-a777-bacc427c11c5" containerName="test-curl-7753-140460067885392" Jul 16 09:36:33.062201 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.062043 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" Jul 16 09:36:33.062201 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.062056 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" Jul 16 09:36:33.062201 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.062065 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" Jul 16 09:36:33.062201 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.062073 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6174c1f2-9670-428e-a777-bacc427c11c5" containerName="test-curl-7753-140460067885392" Jul 16 09:36:33.062201 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.062146 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" Jul 16 09:36:33.062201 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.062173 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="631fb0cc-2136-4bb6-8f63-066d7584d52d" containerName="istio-proxy" Jul 16 09:36:33.062429 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.062269 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="524a0c18-f938-4c76-8833-ce62a32465ea" containerName="istio-proxy" Jul 16 09:36:33.065213 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.065195 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xjqh/must-gather-qj86d" Jul 16 09:36:33.070569 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.070534 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8xjqh\"/\"default-dockercfg-lv2zr\"" Jul 16 09:36:33.073602 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.073579 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xjqh\"/\"kube-root-ca.crt\"" Jul 16 09:36:33.074430 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.074409 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xjqh\"/\"openshift-service-ca.crt\"" Jul 16 09:36:33.085423 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.085401 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xjqh/must-gather-qj86d"] Jul 16 09:36:33.128235 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.128214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bed0c008-1088-4bb8-a0b7-77d92a19c1d0-must-gather-output\") pod \"must-gather-qj86d\" (UID: \"bed0c008-1088-4bb8-a0b7-77d92a19c1d0\") " pod="openshift-must-gather-8xjqh/must-gather-qj86d" Jul 16 09:36:33.128235 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.128239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmldb\" (UniqueName: \"kubernetes.io/projected/bed0c008-1088-4bb8-a0b7-77d92a19c1d0-kube-api-access-kmldb\") pod \"must-gather-qj86d\" (UID: \"bed0c008-1088-4bb8-a0b7-77d92a19c1d0\") " pod="openshift-must-gather-8xjqh/must-gather-qj86d" Jul 16 09:36:33.228964 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.228941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bed0c008-1088-4bb8-a0b7-77d92a19c1d0-must-gather-output\") pod \"must-gather-qj86d\" (UID: \"bed0c008-1088-4bb8-a0b7-77d92a19c1d0\") " pod="openshift-must-gather-8xjqh/must-gather-qj86d" Jul 16 09:36:33.229059 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.228965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmldb\" (UniqueName: \"kubernetes.io/projected/bed0c008-1088-4bb8-a0b7-77d92a19c1d0-kube-api-access-kmldb\") pod \"must-gather-qj86d\" (UID: \"bed0c008-1088-4bb8-a0b7-77d92a19c1d0\") " pod="openshift-must-gather-8xjqh/must-gather-qj86d" Jul 16 09:36:33.229263 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.229244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bed0c008-1088-4bb8-a0b7-77d92a19c1d0-must-gather-output\") pod \"must-gather-qj86d\" (UID: \"bed0c008-1088-4bb8-a0b7-77d92a19c1d0\") " pod="openshift-must-gather-8xjqh/must-gather-qj86d" Jul 16 09:36:33.240241 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.240219 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmldb\" (UniqueName: \"kubernetes.io/projected/bed0c008-1088-4bb8-a0b7-77d92a19c1d0-kube-api-access-kmldb\") pod \"must-gather-qj86d\" (UID: \"bed0c008-1088-4bb8-a0b7-77d92a19c1d0\") " pod="openshift-must-gather-8xjqh/must-gather-qj86d" Jul 16 09:36:33.373889 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.373869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xjqh/must-gather-qj86d" Jul 16 09:36:33.504266 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:33.504231 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xjqh/must-gather-qj86d"] Jul 16 09:36:33.506637 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:36:33.506611 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbed0c008_1088_4bb8_a0b7_77d92a19c1d0.slice/crio-51840d7f04c0366f760ea8b4bd7379b03ab831a53bec7dcb6de01172ffefbdec WatchSource:0}: Error finding container 51840d7f04c0366f760ea8b4bd7379b03ab831a53bec7dcb6de01172ffefbdec: Status 404 returned error can't find the container with id 51840d7f04c0366f760ea8b4bd7379b03ab831a53bec7dcb6de01172ffefbdec Jul 16 09:36:34.486034 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:34.485989 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xjqh/must-gather-qj86d" event={"ID":"bed0c008-1088-4bb8-a0b7-77d92a19c1d0","Type":"ContainerStarted","Data":"19c1087a03efc505942555f3aeb4791cac1167553b6d2d87a9a0132bd9d371e1"} Jul 16 09:36:34.486460 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:34.486050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xjqh/must-gather-qj86d" event={"ID":"bed0c008-1088-4bb8-a0b7-77d92a19c1d0","Type":"ContainerStarted","Data":"51840d7f04c0366f760ea8b4bd7379b03ab831a53bec7dcb6de01172ffefbdec"} Jul 16 09:36:35.493010 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:35.492973 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xjqh/must-gather-qj86d" event={"ID":"bed0c008-1088-4bb8-a0b7-77d92a19c1d0","Type":"ContainerStarted","Data":"90309eef4bf84f32361712b6773bd3fafd0a1f9451a75b9c0982da94b01109fd"} Jul 16 09:36:35.510291 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:35.510243 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8xjqh/must-gather-qj86d" podStartSLOduration=1.682155002 podStartE2EDuration="2.510227679s" totalImagesPullingTime="828.072677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 09:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:36:35.50817739 +0000 UTC m=+2508.882311682" watchObservedRunningTime="2026-07-16 09:36:35.510227679 +0000 UTC m=+2508.884362025" Jul 16 09:36:35.983584 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:35.983550 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pd2px_491e030c-547f-4c64-be98-eb0c8313b800/global-pull-secret-syncer/0.log" Jul 16 09:36:36.103720 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:36.103685 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-njtsp_180ba7da-0767-42a2-9f01-4492fff3b648/konnectivity-agent/0.log" Jul 16 09:36:36.125682 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:36.125655 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-59.ec2.internal_8dfbfc1bca38cfe8c6a6944c393e430e/haproxy/0.log" Jul 16 09:36:41.860306 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:41.860277 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-jl6gr_73e74bc7-8621-4741-ae0e-e325fe735ec4/limitador/0.log" Jul 16 09:36:43.706663 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:43.706629 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-54bw9_936e4c26-9180-4276-ab88-95b85db3653b/cluster-monitoring-operator/0.log" Jul 16 09:36:43.931352 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:43.931318 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px2qw_1e806b6a-741c-4a34-be65-c965ce89ec25/node-exporter/0.log" Jul 16 09:36:43.957109 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:43.957034 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px2qw_1e806b6a-741c-4a34-be65-c965ce89ec25/kube-rbac-proxy/0.log" Jul 16 09:36:43.985282 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:43.985252 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px2qw_1e806b6a-741c-4a34-be65-c965ce89ec25/init-textfile/0.log" Jul 16 09:36:44.096615 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.096587 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n2vfq_240fc7fe-4395-4dbc-b0ea-f9b6da29938e/kube-rbac-proxy-main/0.log" Jul 16 09:36:44.120037 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.119975 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n2vfq_240fc7fe-4395-4dbc-b0ea-f9b6da29938e/kube-rbac-proxy-self/0.log" Jul 16 09:36:44.140695 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.140670 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-n2vfq_240fc7fe-4395-4dbc-b0ea-f9b6da29938e/openshift-state-metrics/0.log" Jul 16 09:36:44.175602 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.175566 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_46e348c6-6e0a-4346-8997-0f1273b4abde/prometheus/0.log" Jul 16 09:36:44.197609 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.197581 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_46e348c6-6e0a-4346-8997-0f1273b4abde/config-reloader/0.log" Jul 16 09:36:44.221019 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.220943 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_46e348c6-6e0a-4346-8997-0f1273b4abde/thanos-sidecar/0.log" Jul 16 09:36:44.243174 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.243129 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_46e348c6-6e0a-4346-8997-0f1273b4abde/kube-rbac-proxy-web/0.log" Jul 16 09:36:44.266509 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.266480 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_46e348c6-6e0a-4346-8997-0f1273b4abde/kube-rbac-proxy/0.log" Jul 16 09:36:44.289394 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.289353 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_46e348c6-6e0a-4346-8997-0f1273b4abde/kube-rbac-proxy-thanos/0.log" Jul 16 09:36:44.313400 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:44.313352 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_46e348c6-6e0a-4346-8997-0f1273b4abde/init-config-reloader/0.log" Jul 16 09:36:45.497997 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.497952 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5"] Jul 16 09:36:45.505513 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.505490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.511114 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.511082 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5"] Jul 16 09:36:45.546896 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.546596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8ms2\" (UniqueName: \"kubernetes.io/projected/8fcd760d-3bd6-4d75-9fca-ef72709f5731-kube-api-access-c8ms2\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.546896 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.546644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-podres\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.546896 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.546687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-proc\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.546896 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.546758 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-sys\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.546896 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.546786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-lib-modules\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.647284 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.647250 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8ms2\" (UniqueName: \"kubernetes.io/projected/8fcd760d-3bd6-4d75-9fca-ef72709f5731-kube-api-access-c8ms2\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.647534 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.647328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-podres\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.647534 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.647383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-proc\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.647534 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.647466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-sys\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.647534 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.647491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-lib-modules\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.647770 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.647527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-podres\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.647770 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.647619 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-proc\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.647770 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.647621 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-sys\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.647770 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.647660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fcd760d-3bd6-4d75-9fca-ef72709f5731-lib-modules\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.656043 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.656015 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8ms2\" (UniqueName: \"kubernetes.io/projected/8fcd760d-3bd6-4d75-9fca-ef72709f5731-kube-api-access-c8ms2\") pod \"perf-node-gather-daemonset-mwxf5\" (UID: \"8fcd760d-3bd6-4d75-9fca-ef72709f5731\") " pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:45.821062 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:45.820979 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:46.005047 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:46.005005 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5"] Jul 16 09:36:46.009755 ip-10-0-128-59 kubenswrapper[2574]: W0716 09:36:46.009714 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8fcd760d_3bd6_4d75_9fca_ef72709f5731.slice/crio-0a1f7234e2a85dff1d6a409a5f74dfd89e53bc55a4388945cd3fbba01bf53a6e WatchSource:0}: Error finding container 0a1f7234e2a85dff1d6a409a5f74dfd89e53bc55a4388945cd3fbba01bf53a6e: Status 404 returned error can't find the container with id 0a1f7234e2a85dff1d6a409a5f74dfd89e53bc55a4388945cd3fbba01bf53a6e Jul 16 09:36:46.564621 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:46.564577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" event={"ID":"8fcd760d-3bd6-4d75-9fca-ef72709f5731","Type":"ContainerStarted","Data":"9f94c55d55ea2fd992ad25095819d4a85cf74e16c733ca35070028ffa0cf2e1a"} Jul 16 09:36:46.565214 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:46.565184 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" event={"ID":"8fcd760d-3bd6-4d75-9fca-ef72709f5731","Type":"ContainerStarted","Data":"0a1f7234e2a85dff1d6a409a5f74dfd89e53bc55a4388945cd3fbba01bf53a6e"} Jul 16 09:36:46.565371 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:46.565232 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:46.879206 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:46.879176 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-tdvbs_2d0c6c73-8026-446d-95bd-e8515a8552e4/download-server/0.log" Jul 16 09:36:48.608827 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:48.608799 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qkrk4_837b276c-808f-46c4-88c2-6c590b5d6c02/dns/0.log" Jul 16 09:36:48.630046 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:48.629994 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qkrk4_837b276c-808f-46c4-88c2-6c590b5d6c02/kube-rbac-proxy/0.log" Jul 16 09:36:48.680260 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:48.680235 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qhwlv_0c64883f-3ead-4052-bea8-c09675421988/dns-node-resolver/0.log" Jul 16 09:36:49.382414 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:49.382387 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lpps8_928dda30-aac8-42e9-a376-9eb125067780/node-ca/0.log" Jul 16 09:36:50.320066 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:50.320030 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4f9phv_977280d7-4a3a-4fcb-87c4-c863fb79626a/istio-proxy/0.log" Jul 16 09:36:50.443484 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:50.443457 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-856d68f954-94qtz_87d37615-95c8-4a58-86bd-6c7ad1410b29/kube-auth-proxy/0.log" Jul 16 09:36:51.341239 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:51.341197 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j6rmb_5abfceca-c1c1-42a0-bc57-ff208b0b63fd/serve-healthcheck-canary/0.log" Jul 16 09:36:51.861189 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:51.861138 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-8g4xv_019c8aea-66ce-4497-bc6a-550bdefad29c/insights-operator/0.log" Jul 16 09:36:51.861705 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:51.861686 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-8g4xv_019c8aea-66ce-4497-bc6a-550bdefad29c/insights-operator/1.log" Jul 16 09:36:51.882965 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:51.882943 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmcmn_90534626-e36c-448a-93f8-159f941e95b8/kube-rbac-proxy/0.log" Jul 16 09:36:51.903553 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:51.903530 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmcmn_90534626-e36c-448a-93f8-159f941e95b8/exporter/0.log" Jul 16 09:36:51.925795 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:51.925772 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bmcmn_90534626-e36c-448a-93f8-159f941e95b8/extractor/0.log" Jul 16 09:36:53.589812 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:53.589783 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" Jul 16 09:36:53.608535 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:53.608481 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8xjqh/perf-node-gather-daemonset-mwxf5" podStartSLOduration=8.608464093 podStartE2EDuration="8.608464093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 09:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 09:36:46.582170526 +0000 UTC m=+2519.956304811" watchObservedRunningTime="2026-07-16 09:36:53.608464093 +0000 UTC m=+2526.982598393" Jul 16 09:36:54.768200 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:54.768147 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b875787f8-78b79_d8b8fd9d-aebe-4607-b178-df74a27c1d23/manager/0.log" Jul 16 09:36:54.792723 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:54.792693 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b875787f8-78b79_d8b8fd9d-aebe-4607-b178-df74a27c1d23/copy-manifests/0.log" Jul 16 09:36:55.599096 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:55.599052 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4f9phv_977280d7-4a3a-4fcb-87c4-c863fb79626a/istio-proxy/0.log" Jul 16 09:36:55.712979 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:55.712937 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-856d68f954-94qtz_87d37615-95c8-4a58-86bd-6c7ad1410b29/kube-auth-proxy/0.log" Jul 16 09:36:56.465401 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:56.465368 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74556d96d9-xgdgc_a516d268-427a-400a-8911-9625fe05627f/manager/0.log" Jul 16 09:36:57.250584 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:57.250546 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4f9phv_977280d7-4a3a-4fcb-87c4-c863fb79626a/istio-proxy/0.log" Jul 16 09:36:57.362062 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:36:57.362030 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-856d68f954-94qtz_87d37615-95c8-4a58-86bd-6c7ad1410b29/kube-auth-proxy/0.log" Jul 16 09:37:01.945586 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:01.945550 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-j4c9z_4526fa9c-78bc-463d-8244-cf63b0c783bb/kube-storage-version-migrator-operator/1.log" Jul 16 09:37:01.946418 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:01.946396 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-j4c9z_4526fa9c-78bc-463d-8244-cf63b0c783bb/kube-storage-version-migrator-operator/0.log" Jul 16 09:37:03.045118 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.045086 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6j42v_c7b6480a-2e28-4325-b994-4a1bd5516125/kube-multus/0.log" Jul 16 09:37:03.068802 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.068776 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zqbr_9dac8b01-f667-4faa-801b-c69c9655c829/kube-multus-additional-cni-plugins/0.log" Jul 16 09:37:03.090914 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.090886 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zqbr_9dac8b01-f667-4faa-801b-c69c9655c829/egress-router-binary-copy/0.log" Jul 16 09:37:03.112469 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.112448 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zqbr_9dac8b01-f667-4faa-801b-c69c9655c829/cni-plugins/0.log" Jul 16 09:37:03.134510 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.134492 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zqbr_9dac8b01-f667-4faa-801b-c69c9655c829/bond-cni-plugin/0.log" Jul 16 09:37:03.155563 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.155540 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zqbr_9dac8b01-f667-4faa-801b-c69c9655c829/routeoverride-cni/0.log" Jul 16 09:37:03.177572 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.177545 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zqbr_9dac8b01-f667-4faa-801b-c69c9655c829/whereabouts-cni-bincopy/0.log" Jul 16 09:37:03.198622 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.198604 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2zqbr_9dac8b01-f667-4faa-801b-c69c9655c829/whereabouts-cni/0.log" Jul 16 09:37:03.627657 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.627629 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7zblf_7689dfa7-1f38-485d-a5e5-1fbea1c74f57/network-metrics-daemon/0.log" Jul 16 09:37:03.663363 ip-10-0-128-59 kubenswrapper[2574]: I0716 09:37:03.663338 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7zblf_7689dfa7-1f38-485d-a5e5-1fbea1c74f57/kube-rbac-proxy/0.log"