Jul 20 04:57:32.423142 ip-10-0-133-196 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 20 04:57:32.423158 ip-10-0-133-196 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 20 04:57:32.423168 ip-10-0-133-196 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 20 04:57:32.423442 ip-10-0-133-196 systemd[1]: Failed to start Kubernetes Kubelet. Jul 20 04:57:42.632952 ip-10-0-133-196 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 20 04:57:42.632975 ip-10-0-133-196 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 45ea24b2c1354a4c8062228b52dc723f -- Jul 20 04:59:29.496681 ip-10-0-133-196 systemd[1]: Starting Kubernetes Kubelet... Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.829559 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833676 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833690 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833694 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833698 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833701 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833705 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833708 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 04:59:29.939025 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833711 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 04:59:29.899818 ip-10-0-133-196 systemd[1]: Started Kubernetes Kubelet. Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833713 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833716 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833719 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833725 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833728 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833731 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833734 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833736 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833739 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833742 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833745 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833747 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833750 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833753 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833765 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833768 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833771 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833773 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 04:59:29.941794 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833776 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833779 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833781 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833784 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833786 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833789 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833791 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833794 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833796 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833799 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833801 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833805 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833808 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833810 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833813 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833815 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833818 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833823 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833826 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 04:59:29.943684 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833829 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833832 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833835 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833838 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833841 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833843 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833846 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833849 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833852 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833855 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833857 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833860 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833863 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833865 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833867 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833870 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833872 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833875 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833878 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833880 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 04:59:29.945918 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833884 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833889 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833892 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833895 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833898 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833901 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833904 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833907 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833910 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833912 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833916 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833918 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833921 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833924 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833926 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833929 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833931 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833933 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833936 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833938 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 04:59:29.947823 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833942 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833945 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833948 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833953 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833956 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833958 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833961 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833963 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833965 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833968 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833971 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833973 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833976 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.833978 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834698 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834704 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834708 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834710 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834713 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 04:59:29.950210 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834716 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834718 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834721 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834724 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834728 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834731 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834735 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834739 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834743 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834746 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834750 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834753 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834756 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834758 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834761 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834764 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834769 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834772 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834775 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 04:59:29.952081 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834778 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834781 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834783 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834786 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834789 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834791 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834794 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834796 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834799 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834803 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834806 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834810 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834812 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834815 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834818 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834821 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834823 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834826 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834830 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834832 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 04:59:29.954536 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834835 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834837 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834840 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834842 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834845 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834847 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834850 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834852 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834854 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834858 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834860 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834863 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834866 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834868 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834871 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834873 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834876 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834878 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834881 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834883 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 04:59:29.955194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834886 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834888 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834891 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834893 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834896 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834901 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834904 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834906 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834909 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834911 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834914 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834917 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834919 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834922 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834924 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834926 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834929 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834931 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834934 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834936 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 04:59:29.956068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834939 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834943 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834945 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834947 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834950 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834952 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834955 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834957 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834960 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834962 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834965 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834967 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834970 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834972 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.834975 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835335 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835339 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835344 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835347 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835349 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 20 04:59:29.956948 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835352 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835355 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835357 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835375 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835378 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835381 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835384 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835386 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835389 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835391 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835394 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835397 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835399 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835403 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835406 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835408 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835411 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835413 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835416 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835419 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 04:59:29.957618 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835421 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835424 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835427 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835429 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835432 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835434 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835437 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835439 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835442 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835445 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835447 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835450 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835453 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835455 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835458 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835460 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835464 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835466 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835469 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835472 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 04:59:29.958274 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835474 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835477 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835479 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835482 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835485 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835489 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835492 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835494 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835497 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835499 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835501 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835504 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835507 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835509 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835512 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835514 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835518 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835520 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835523 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835526 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 04:59:29.958973 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835528 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835532 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835536 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835540 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835545 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835551 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835555 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835559 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835563 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835568 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835572 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835576 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835580 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835584 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835588 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835591 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835594 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835599 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835601 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 04:59:29.959858 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835604 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835607 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835610 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835613 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835615 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835618 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835620 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835623 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835626 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835628 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835631 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835633 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835636 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835638 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.835642 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835721 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835731 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835741 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835747 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835754 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835759 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 20 04:59:29.960551 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835766 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835773 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835778 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835781 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835785 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835788 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835791 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835794 2579 flags.go:64] FLAG: --cgroup-root="" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835797 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835802 2579 flags.go:64] FLAG: --client-ca-file="" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835805 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835808 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835812 2579 flags.go:64] FLAG: --cluster-domain="" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835815 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835818 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835821 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835825 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835829 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835832 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835835 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835838 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835841 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835844 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835847 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 20 04:59:29.961241 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835850 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835856 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835860 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835863 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835866 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835869 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835872 2579 flags.go:64] FLAG: --enable-server="true" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835876 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835881 2579 flags.go:64] FLAG: --event-burst="100" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835884 2579 flags.go:64] FLAG: --event-qps="50" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835887 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835891 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835894 2579 flags.go:64] FLAG: --eviction-hard="" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835898 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835901 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835904 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835906 2579 flags.go:64] FLAG: --eviction-soft="" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835911 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835914 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835917 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835919 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835922 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835925 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835929 2579 flags.go:64] FLAG: --feature-gates="" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835932 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 20 04:59:29.962773 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835936 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835938 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835941 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835945 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835948 2579 flags.go:64] FLAG: --help="false" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835951 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-196.ec2.internal" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835954 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835957 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835961 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835964 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835967 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835970 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835973 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835976 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835979 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835982 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835985 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835988 2579 flags.go:64] FLAG: --kube-reserved="" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835990 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835994 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.835997 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836000 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836002 2579 flags.go:64] FLAG: --lock-file="" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836005 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 20 04:59:29.964679 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836009 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836012 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836018 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836020 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836023 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836026 2579 flags.go:64] FLAG: --logging-format="text" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836029 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836032 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836035 2579 flags.go:64] FLAG: --manifest-url="" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836038 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836042 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836045 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836049 2579 flags.go:64] FLAG: --max-pods="110" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836052 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836055 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836058 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836062 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836065 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836068 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836070 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836077 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836081 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836084 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 20 04:59:29.965976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836087 2579 flags.go:64] FLAG: --pod-cidr="" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836089 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836095 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836098 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836101 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836104 2579 flags.go:64] FLAG: --port="10250" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836107 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836110 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-009d26b01091b3ed7" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836114 2579 flags.go:64] FLAG: --qos-reserved="" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836118 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836121 2579 flags.go:64] FLAG: --register-node="true" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836124 2579 flags.go:64] FLAG: --register-with-taints="" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836128 2579 flags.go:64] FLAG: --registry-burst="10" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836131 2579 flags.go:64] FLAG: --registry-qps="5" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836134 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836137 2579 flags.go:64] FLAG: --reserved-memory="" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836140 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836143 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836147 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836150 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836152 2579 flags.go:64] FLAG: --runonce="false" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836155 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836158 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836162 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836164 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 20 04:59:29.978976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836167 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836172 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836175 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836178 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836181 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836184 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836186 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836189 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836192 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836195 2579 flags.go:64] FLAG: --system-cgroups="" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836198 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836203 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836206 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836209 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836213 2579 flags.go:64] FLAG: --tls-min-version="" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836216 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836220 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836223 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836226 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836229 2579 flags.go:64] FLAG: --v="2" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836233 2579 flags.go:64] FLAG: --version="false" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836237 2579 flags.go:64] FLAG: --vmodule="" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836247 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.836250 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836388 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 04:59:29.980674 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836392 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836396 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836399 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836401 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836404 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836407 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836410 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836413 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836417 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836420 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836423 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836426 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836430 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836434 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836437 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836440 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836442 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836445 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836447 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 04:59:29.981759 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836450 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836453 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836456 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836459 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836464 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836466 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836469 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836471 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836474 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836476 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836479 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836481 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836484 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836486 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836489 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836491 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836494 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836496 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836499 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836501 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 04:59:29.982617 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836504 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836507 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836510 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836513 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836515 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836518 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836520 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836523 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836525 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836528 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836531 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836533 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836536 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836538 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836541 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836550 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836555 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836557 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836560 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836563 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 04:59:29.983490 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836565 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836567 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836570 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836573 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836576 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836580 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836583 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836586 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836588 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836591 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836594 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836596 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836599 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836603 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836606 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836608 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836611 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836613 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836616 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836619 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 04:59:29.984334 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836621 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836624 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836627 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836629 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836632 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836635 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836637 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836639 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836644 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836646 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836649 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836651 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836656 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836659 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836662 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836664 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836667 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836670 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 04:59:29.985213 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.836673 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.837418 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.844883 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.844905 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.844934 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.844943 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845020 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845028 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845034 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845038 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845042 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845046 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845050 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845054 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 04:59:29.986061 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845059 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845065 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845069 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845073 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845077 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845081 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845085 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845088 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845091 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845095 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845099 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845103 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845108 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845111 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845115 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845118 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845122 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845126 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845130 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845135 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 04:59:29.986910 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845141 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845146 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845151 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845154 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845158 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845161 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845165 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845168 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845172 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845176 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845180 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845184 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845187 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845191 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845194 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845198 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845202 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845206 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845209 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845213 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 04:59:29.987755 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845216 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845221 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845224 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845228 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845232 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845235 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845239 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845242 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845247 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845251 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845254 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845262 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845267 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845271 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845275 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845279 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845283 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845287 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845290 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845295 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 04:59:29.988639 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845298 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845302 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845306 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845310 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845314 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845318 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845324 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845328 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845333 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845336 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845341 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845345 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845350 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845354 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845357 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845378 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845383 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845387 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845391 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845394 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 04:59:29.989497 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845397 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845401 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845405 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845410 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845416 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845419 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845423 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845431 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845436 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845440 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845444 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.845451 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845612 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845620 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 20 04:59:29.990352 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845624 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845628 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845632 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845636 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845640 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845644 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845647 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845651 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845655 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845659 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845663 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845667 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845671 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845675 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845679 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845683 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845687 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845692 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845696 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 20 04:59:29.991194 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845700 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845704 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845707 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845714 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845718 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845722 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845726 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845730 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845735 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845738 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845742 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845746 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845750 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845753 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845759 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845764 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845769 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845772 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845776 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845780 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 20 04:59:29.992068 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845784 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845787 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845790 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845794 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845797 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845801 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845805 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845808 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845812 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845815 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845819 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845822 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845826 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845829 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845833 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845839 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845843 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845847 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845850 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845855 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 20 04:59:29.992938 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845865 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845868 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845872 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845876 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845880 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845884 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845887 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845891 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845894 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845897 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845901 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845905 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845909 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845913 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845916 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845919 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845923 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845927 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845930 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845934 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 20 04:59:30.130723 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845937 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845940 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845944 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845948 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845951 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845955 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845958 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845965 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845969 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845973 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845976 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845980 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845984 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845988 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845992 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.845996 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.846000 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 20 04:59:30.152580 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:29.846004 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.846011 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.846167 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.849910 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.851125 2579 server.go:1019] "Starting client certificate rotation" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.851228 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.851267 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.855055 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.856258 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.864335 2579 log.go:25] "Validated CRI v1 runtime API" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.867501 2579 log.go:25] "Validated CRI v1 image API" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.871067 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 20 04:59:30.153425 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.873157 2579 fs.go:135] Filesystem UUIDs: map[3a3ef158-0688-4036-a1b4-04b90c33570e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ab72532b-4ea4-414f-bc68-c49ffe5180be:/dev/nvme0n1p3] Jul 20 04:59:30.153935 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.873176 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 20 04:59:30.157548 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.878425 2579 manager.go:216] Machine: {Timestamp:2026-07-20 04:59:29.877130374 +0000 UTC m=+0.276307082 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101009 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a7a6229908a08674f8c9ef7e641ab SystemUUID:ec2a7a62-2990-8a08-674f-8c9ef7e641ab BootID:45ea24b2-c135-4a4c-8062-228b52dc723f Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:66:b9:b3:4d:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:66:b9:b3:4d:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:3b:12:e1:86:62 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 20 04:59:30.157548 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.878542 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 20 04:59:30.157548 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.878631 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 20 04:59:30.157548 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.880195 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 04:59:30.157548 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.880984 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 20 04:59:30.157548 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.881014 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-196.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 20 04:59:30.157548 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.882325 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.882338 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.882345 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.882923 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.884290 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.888109 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.888225 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.888254 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.888264 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.888278 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.888288 2579 kubelet.go:398] "Adding apiserver pod source" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.888300 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.888692 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.888708 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.891581 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.893464 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.893501 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894147 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894164 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894173 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894180 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894193 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894201 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894214 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894222 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894231 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894239 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894251 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894259 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894882 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.894893 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.898941 2579 server.go:1268] "Started kubelet" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.899029 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.899034 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.899112 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.900288 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.901836 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-196.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:29.902317 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:29.902320 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.902973 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 20 04:59:30.157959 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.903720 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b7j9f" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:29.905630 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:29.906864 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-196.ec2.internal.18c3e6bb6522e10b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-196.ec2.internal,UID:ip-10-0-133-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-196.ec2.internal,},FirstTimestamp:2026-07-20 04:59:29.898905867 +0000 UTC m=+0.298082579,LastTimestamp:2026-07-20 04:59:29.898905867 +0000 UTC m=+0.298082579,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-196.ec2.internal,}" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.908593 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.908608 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:29.908664 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.909199 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.909263 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.909277 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.909394 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.909403 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.909924 2579 factory.go:55] Registering systemd factory Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.909952 2579 factory.go:223] Registration of the systemd container factory successfully Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.910154 2579 factory.go:153] Registering CRI-O factory Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.910165 2579 factory.go:223] Registration of the crio container factory successfully Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.910211 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.910231 2579 factory.go:103] Registering Raw factory Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.910241 2579 manager.go:1196] Started watching for new ooms in manager Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.910669 2579 manager.go:349] Starting recovery of all containers Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.911488 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b7j9f" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:29.911584 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:29.911741 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.923534 2579 manager.go:354] Recovery completed Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.928813 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.930837 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientMemory" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.930874 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.930889 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientPID" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.931520 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.931532 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.931538 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.931549 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.932354 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientMemory" Jul 20 04:59:30.161355 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.932408 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.932424 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientPID" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.933628 2579 policy_none.go:49] "None policy: Start" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.933641 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.933651 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:29.935646 2579 policy_none.go:47] "Start" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.000081 2579 manager.go:337] "Starting Device Plugin manager" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.000293 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.000309 2579 server.go:87] "Starting device plugin registration server" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.000595 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.000609 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.000730 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.000817 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.000827 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.001462 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.001512 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.017790 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.019011 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.019053 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.019075 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.019107 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.021267 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.101181 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.107808 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientMemory" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.107844 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.107855 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientPID" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.107879 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.115467 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.115487 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-196.ec2.internal\": node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.119300 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal"] Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.119357 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.120814 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientMemory" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.120838 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.120848 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientPID" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.123050 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.123163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.123194 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.123802 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientMemory" Jul 20 04:59:30.162321 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.123831 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.123847 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientPID" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.124225 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientMemory" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.124253 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.124266 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientPID" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.126513 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.126542 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.128654 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientMemory" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.128681 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasNoDiskPressure" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.128692 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeHasSufficientPID" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.130673 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.151221 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-196.ec2.internal\" not found" node="ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.163386 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.154655 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-196.ec2.internal\" not found" node="ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.211352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dc92cae0202c6293dc0e89c9e7409eef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal\" (UID: \"dc92cae0202c6293dc0e89c9e7409eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.211409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc92cae0202c6293dc0e89c9e7409eef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal\" (UID: \"dc92cae0202c6293dc0e89c9e7409eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.211452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94f9424feb20d8f7b53d0fbc4a5441ee-config\") pod \"kube-apiserver-proxy-ip-10-0-133-196.ec2.internal\" (UID: \"94f9424feb20d8f7b53d0fbc4a5441ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.231444 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.311826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dc92cae0202c6293dc0e89c9e7409eef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal\" (UID: \"dc92cae0202c6293dc0e89c9e7409eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.311859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc92cae0202c6293dc0e89c9e7409eef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal\" (UID: \"dc92cae0202c6293dc0e89c9e7409eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.311877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94f9424feb20d8f7b53d0fbc4a5441ee-config\") pod \"kube-apiserver-proxy-ip-10-0-133-196.ec2.internal\" (UID: \"94f9424feb20d8f7b53d0fbc4a5441ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.311928 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/94f9424feb20d8f7b53d0fbc4a5441ee-config\") pod \"kube-apiserver-proxy-ip-10-0-133-196.ec2.internal\" (UID: \"94f9424feb20d8f7b53d0fbc4a5441ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.311918 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dc92cae0202c6293dc0e89c9e7409eef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal\" (UID: \"dc92cae0202c6293dc0e89c9e7409eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:30.311938 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc92cae0202c6293dc0e89c9e7409eef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal\" (UID: \"dc92cae0202c6293dc0e89c9e7409eef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" Jul 20 04:59:30.348360 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:30.331896 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:32.566135 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:32.566036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" Jul 20 04:59:32.566135 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:32.566046 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 04:54:31 +0000 UTC" deadline="2028-01-04 15:39:18.617265251 +0000 UTC" Jul 20 04:59:32.566135 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:32.566113 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12802h39m46.051156168s" Jul 20 04:59:32.566782 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:32.566163 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:32.566782 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:32.566046 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal" Jul 20 04:59:32.566782 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:32.566175 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 20 04:59:32.566782 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:32.566299 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding" Jul 20 04:59:32.567022 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:32.567006 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: Get "https://a62dad94539ad4b2f913fcf4a92c496c-0fa8a40ce9fbecab.elb.us-east-1.amazonaws.com:6443/apis/storage.k8s.io/v1/csinodes/ip-10-0-133-196.ec2.internal?resourceVersion=0": http2: client connection force closed via ClientConn.Close Jul 20 04:59:32.578509 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:32.578490 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 04:59:32.666678 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:32.666640 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:32.767112 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:32.767073 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:32.867560 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:32.867482 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:32.967964 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:32.967930 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:33.068779 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:33.068751 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:33.101227 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:33.101184 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc92cae0202c6293dc0e89c9e7409eef.slice/crio-70cef87d5802f748c3becc24f2c2a7068bc8f8b2a15b8fd400f71fde66137f24 WatchSource:0}: Error finding container 70cef87d5802f748c3becc24f2c2a7068bc8f8b2a15b8fd400f71fde66137f24: Status 404 returned error can't find the container with id 70cef87d5802f748c3becc24f2c2a7068bc8f8b2a15b8fd400f71fde66137f24 Jul 20 04:59:33.101679 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:33.101652 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f9424feb20d8f7b53d0fbc4a5441ee.slice/crio-90c519c29ba81fae0bf6cbd0f8b204492d96b0bcf0b543e88f372dd02b6f61db WatchSource:0}: Error finding container 90c519c29ba81fae0bf6cbd0f8b204492d96b0bcf0b543e88f372dd02b6f61db: Status 404 returned error can't find the container with id 90c519c29ba81fae0bf6cbd0f8b204492d96b0bcf0b543e88f372dd02b6f61db Jul 20 04:59:33.106323 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:33.106306 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 04:59:33.168913 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:33.168818 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:33.269228 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:33.269192 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:33.369696 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:33.369643 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:33.470183 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:33.470104 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-196.ec2.internal\" not found" Jul 20 04:59:33.569541 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:33.569384 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 04:59:33.569541 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:33.569438 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 04:59:33.570316 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:33.569916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal" event={"ID":"94f9424feb20d8f7b53d0fbc4a5441ee","Type":"ContainerStarted","Data":"90c519c29ba81fae0bf6cbd0f8b204492d96b0bcf0b543e88f372dd02b6f61db"} Jul 20 04:59:33.571913 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:33.571885 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" event={"ID":"dc92cae0202c6293dc0e89c9e7409eef","Type":"ContainerStarted","Data":"70cef87d5802f748c3becc24f2c2a7068bc8f8b2a15b8fd400f71fde66137f24"} Jul 20 04:59:33.639647 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:33.639398 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" Jul 20 04:59:33.650510 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:33.650477 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 04:59:33.651443 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:33.651415 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal" Jul 20 04:59:33.656901 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:33.656879 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 20 04:59:34.082340 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.082303 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 20 04:59:34.567376 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.567303 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 20 04:59:34.567520 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.567323 2579 apiserver.go:52] "Watching apiserver" Jul 20 04:59:34.574955 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.574927 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 20 04:59:34.575954 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.575933 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-5gnfx","openshift-multus/multus-vt4vk","openshift-network-diagnostics/network-check-target-7gpl4","openshift-dns/node-resolver-jv8n9","openshift-image-registry/node-ca-h2rbz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal","openshift-multus/network-metrics-daemon-krtbd","openshift-network-operator/iptables-alerter-vrcx8","openshift-ovn-kubernetes/ovnkube-node-bnv7f","kube-system/global-pull-secret-syncer-c7ls5","kube-system/konnectivity-agent-fpdbr","kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v","openshift-cluster-node-tuning-operator/tuned-gwx25"] Jul 20 04:59:34.577714 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.577694 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 20 04:59:34.578578 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.578558 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:34.578673 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.578639 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:34.580756 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.580733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.583019 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.582835 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 20 04:59:34.583142 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.583083 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:34.583196 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.583145 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:34.583249 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.583195 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ns7pw\"" Jul 20 04:59:34.583300 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.583248 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 20 04:59:34.584454 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.584423 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 20 04:59:34.585160 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.585142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.587586 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.587345 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 20 04:59:34.587586 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.587408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.587728 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.587676 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 20 04:59:34.587728 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.587688 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7j556\"" Jul 20 04:59:34.589582 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.589554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.589677 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.589660 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 20 04:59:34.589979 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.589960 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 20 04:59:34.590157 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.589978 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 20 04:59:34.590157 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.590044 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8bzk2\"" Jul 20 04:59:34.591711 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.591680 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 20 04:59:34.591807 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.591735 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g5vp8\"" Jul 20 04:59:34.592052 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.591969 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 20 04:59:34.592052 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.591996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.594317 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.594297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.594573 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.594551 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 20 04:59:34.595339 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.595301 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 20 04:59:34.595339 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.595327 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 20 04:59:34.595491 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.595303 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 20 04:59:34.595833 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.595816 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 20 04:59:34.595912 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.595890 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 20 04:59:34.596009 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.595992 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jqzrx\"" Jul 20 04:59:34.596717 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.596698 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 20 04:59:34.597011 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.596981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.597652 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.597156 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bq5lz\"" Jul 20 04:59:34.597652 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.597394 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sw2kv" Jul 20 04:59:34.598534 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.598010 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 20 04:59:34.598534 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.598121 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 20 04:59:34.598534 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.598009 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 20 04:59:34.598831 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.598812 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 20 04:59:34.599665 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.599562 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 20 04:59:34.599806 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.599695 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-64gwh\"" Jul 20 04:59:34.601110 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.601041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:34.601205 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.601127 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:34.602617 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.602599 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sw2kv" Jul 20 04:59:34.603829 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.603807 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:34.605908 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.605887 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 20 04:59:34.605999 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.605930 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 20 04:59:34.606159 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.606137 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mxj6s\"" Jul 20 04:59:34.606255 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.606212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.608369 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.608353 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z4fdg\"" Jul 20 04:59:34.608464 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.608388 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 20 04:59:34.608464 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.608392 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 20 04:59:34.608569 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.608394 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 20 04:59:34.639954 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.639936 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 20 04:59:34.675620 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2nh\" (UniqueName: \"kubernetes.io/projected/90688000-32fe-4a0b-8dc3-32e98e1fd32c-kube-api-access-8w2nh\") pod \"node-ca-h2rbz\" (UID: \"90688000-32fe-4a0b-8dc3-32e98e1fd32c\") " pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.675796 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-lib-modules\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.675796 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fa014da-b395-4e6a-8804-5d3217305e99-cni-binary-copy\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.675796 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.675796 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fa014da-b395-4e6a-8804-5d3217305e99-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.675796 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnj5h\" (UniqueName: \"kubernetes.io/projected/8e5b3021-37f7-4153-bb80-758fb934948c-kube-api-access-rnj5h\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:34.675796 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-run-systemd\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.675796 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-etc-openvswitch\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.675796 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6341cb8a-164f-4125-8aae-b7065e0d7b10-ovnkube-config\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6341cb8a-164f-4125-8aae-b7065e0d7b10-env-overrides\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675826 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-systemd-units\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-run-multus-certs\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-host\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-var-lib-cni-bin\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.675969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-kubernetes\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-sys-fs\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676099 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90688000-32fe-4a0b-8dc3-32e98e1fd32c-host\") pod \"node-ca-h2rbz\" (UID: \"90688000-32fe-4a0b-8dc3-32e98e1fd32c\") " pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676119 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-sysconfig\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-var-lib-kubelet\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.676168 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-run-openvswitch\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-cni-netd\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-var-lib-cni-multus\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7dc30f2a-cb78-4b51-876b-9f06d70cbcc6-iptables-alerter-script\") pod \"iptables-alerter-vrcx8\" (UID: \"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6\") " pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-device-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-cnibin\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21f9b4ba-dead-455b-ac3f-78d65f54ef30-cni-binary-copy\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-hostroot\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgm2h\" (UniqueName: \"kubernetes.io/projected/21f9b4ba-dead-455b-ac3f-78d65f54ef30-kube-api-access-jgm2h\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-modprobe-d\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:34.676723 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-registration-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ksvc\" (UniqueName: \"kubernetes.io/projected/20d23e7e-85a1-4f3f-80a9-4061a4109992-kube-api-access-6ksvc\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-slash\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-cni-dir\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-run\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676682 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-cnibin\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676711 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pd8m\" (UniqueName: \"kubernetes.io/projected/5fa014da-b395-4e6a-8804-5d3217305e99-kube-api-access-6pd8m\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/825150de-fdc4-4920-a483-b648d894dffc-hosts-file\") pod \"node-resolver-jv8n9\" (UID: \"825150de-fdc4-4920-a483-b648d894dffc\") " pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-var-lib-openvswitch\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-socket-dir-parent\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90688000-32fe-4a0b-8dc3-32e98e1fd32c-serviceca\") pod \"node-ca-h2rbz\" (UID: \"90688000-32fe-4a0b-8dc3-32e98e1fd32c\") " pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-sysctl-conf\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-sys\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.677291 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-tmp\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbc8\" (UniqueName: \"kubernetes.io/projected/6341cb8a-164f-4125-8aae-b7065e0d7b10-kube-api-access-kpbc8\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.676996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-system-cni-dir\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-var-lib-kubelet\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rcl\" (UniqueName: \"kubernetes.io/projected/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-kube-api-access-t6rcl\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96dcb202-5818-45a8-a772-b148517779f2-agent-certs\") pod \"konnectivity-agent-fpdbr\" (UID: \"96dcb202-5818-45a8-a772-b148517779f2\") " pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-etc-selinux\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677145 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8flg\" (UniqueName: \"kubernetes.io/projected/825150de-fdc4-4920-a483-b648d894dffc-kube-api-access-q8flg\") pod \"node-resolver-jv8n9\" (UID: \"825150de-fdc4-4920-a483-b648d894dffc\") " pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6341cb8a-164f-4125-8aae-b7065e0d7b10-ovnkube-script-lib\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-run-k8s-cni-cncf-io\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-etc-kubernetes\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-systemd\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-os-release\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-dbus\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:34.677793 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-daemon-config\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx86g\" (UniqueName: \"kubernetes.io/projected/7dc30f2a-cb78-4b51-876b-9f06d70cbcc6-kube-api-access-kx86g\") pod \"iptables-alerter-vrcx8\" (UID: \"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6\") " pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fa014da-b395-4e6a-8804-5d3217305e99-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825150de-fdc4-4920-a483-b648d894dffc-tmp-dir\") pod \"node-resolver-jv8n9\" (UID: \"825150de-fdc4-4920-a483-b648d894dffc\") " pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-node-log\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-os-release\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-run-netns\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dc30f2a-cb78-4b51-876b-9f06d70cbcc6-host-slash\") pod \"iptables-alerter-vrcx8\" (UID: \"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6\") " pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677533 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96dcb202-5818-45a8-a772-b148517779f2-konnectivity-ca\") pod \"konnectivity-agent-fpdbr\" (UID: \"96dcb202-5818-45a8-a772-b148517779f2\") " pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-run-ovn\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-log-socket\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-cni-bin\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-tuned\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-kubelet-config\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:34.678353 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-kubelet\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.678819 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6341cb8a-164f-4125-8aae-b7065e0d7b10-ovn-node-metrics-cert\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.678819 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-sysctl-d\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.678819 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-system-cni-dir\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.678819 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-socket-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.678819 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-run-netns\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.678819 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.677850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-conf-dir\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.778558 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-os-release\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.778558 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-dbus\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:34.778785 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-daemon-config\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.778785 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx86g\" (UniqueName: \"kubernetes.io/projected/7dc30f2a-cb78-4b51-876b-9f06d70cbcc6-kube-api-access-kx86g\") pod \"iptables-alerter-vrcx8\" (UID: \"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6\") " pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.778785 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fa014da-b395-4e6a-8804-5d3217305e99-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.778785 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825150de-fdc4-4920-a483-b648d894dffc-tmp-dir\") pod \"node-resolver-jv8n9\" (UID: \"825150de-fdc4-4920-a483-b648d894dffc\") " pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.778785 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-os-release\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.778785 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778726 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-node-log\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.778785 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-node-log\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.778785 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-os-release\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-run-netns\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dc30f2a-cb78-4b51-876b-9f06d70cbcc6-host-slash\") pod \"iptables-alerter-vrcx8\" (UID: \"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6\") " pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778824 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-dbus\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96dcb202-5818-45a8-a772-b148517779f2-konnectivity-ca\") pod \"konnectivity-agent-fpdbr\" (UID: \"96dcb202-5818-45a8-a772-b148517779f2\") " pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-run-ovn\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-log-socket\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778894 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7dc30f2a-cb78-4b51-876b-9f06d70cbcc6-host-slash\") pod \"iptables-alerter-vrcx8\" (UID: \"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6\") " pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-cni-bin\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778913 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-run-netns\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778958 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-cni-bin\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778956 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-os-release\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.778979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-run-ovn\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779004 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-log-socket\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-tuned\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.779183 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-kubelet-config\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-kubelet\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6341cb8a-164f-4125-8aae-b7065e0d7b10-ovn-node-metrics-cert\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779140 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-sysctl-d\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-system-cni-dir\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779177 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-kubelet-config\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-socket-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-run-netns\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-conf-dir\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2nh\" (UniqueName: \"kubernetes.io/projected/90688000-32fe-4a0b-8dc3-32e98e1fd32c-kube-api-access-8w2nh\") pod \"node-ca-h2rbz\" (UID: \"90688000-32fe-4a0b-8dc3-32e98e1fd32c\") " pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-lib-modules\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-daemon-config\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779303 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5fa014da-b395-4e6a-8804-5d3217305e99-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779361 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-kubelet\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.779715 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779475 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-socket-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779498 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-sysctl-d\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96dcb202-5818-45a8-a772-b148517779f2-konnectivity-ca\") pod \"konnectivity-agent-fpdbr\" (UID: \"96dcb202-5818-45a8-a772-b148517779f2\") " pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-system-cni-dir\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-conf-dir\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-run-netns\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fa014da-b395-4e6a-8804-5d3217305e99-cni-binary-copy\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779683 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-lib-modules\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fa014da-b395-4e6a-8804-5d3217305e99-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnj5h\" (UniqueName: \"kubernetes.io/projected/8e5b3021-37f7-4153-bb80-758fb934948c-kube-api-access-rnj5h\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779785 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5fa014da-b395-4e6a-8804-5d3217305e99-cni-binary-copy\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-run-systemd\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/825150de-fdc4-4920-a483-b648d894dffc-tmp-dir\") pod \"node-resolver-jv8n9\" (UID: \"825150de-fdc4-4920-a483-b648d894dffc\") " pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.780412 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-etc-openvswitch\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6341cb8a-164f-4125-8aae-b7065e0d7b10-ovnkube-config\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6341cb8a-164f-4125-8aae-b7065e0d7b10-env-overrides\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-run-systemd\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779903 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-etc-openvswitch\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-systemd-units\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-run-multus-certs\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779978 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-systemd-units\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.779990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-run-multus-certs\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-host\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780046 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-host\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-var-lib-cni-bin\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-kubernetes\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.780993 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.780149 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-sys-fs\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780166 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-var-lib-cni-bin\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90688000-32fe-4a0b-8dc3-32e98e1fd32c-host\") pod \"node-ca-h2rbz\" (UID: \"90688000-32fe-4a0b-8dc3-32e98e1fd32c\") " pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-sysconfig\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-var-lib-kubelet\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5fa014da-b395-4e6a-8804-5d3217305e99-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-run-openvswitch\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-cni-netd\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-sys-fs\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-var-lib-cni-multus\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-sysconfig\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.781721 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780270 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-var-lib-kubelet\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.780347 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs podName:8e5b3021-37f7-4153-bb80-758fb934948c nodeName:}" failed. No retries permitted until 2026-07-20 04:59:35.28032568 +0000 UTC m=+4.549809351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs") pod "network-metrics-daemon-krtbd" (UID: "8e5b3021-37f7-4153-bb80-758fb934948c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-var-lib-cni-multus\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780371 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-run-openvswitch\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780393 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6341cb8a-164f-4125-8aae-b7065e0d7b10-env-overrides\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7dc30f2a-cb78-4b51-876b-9f06d70cbcc6-iptables-alerter-script\") pod \"iptables-alerter-vrcx8\" (UID: \"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6\") " pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-cni-netd\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6341cb8a-164f-4125-8aae-b7065e0d7b10-ovnkube-config\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-device-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-kubernetes\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-device-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780468 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-cnibin\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780481 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90688000-32fe-4a0b-8dc3-32e98e1fd32c-host\") pod \"node-ca-h2rbz\" (UID: \"90688000-32fe-4a0b-8dc3-32e98e1fd32c\") " pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21f9b4ba-dead-455b-ac3f-78d65f54ef30-cni-binary-copy\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.782403 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-hostroot\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgm2h\" (UniqueName: \"kubernetes.io/projected/21f9b4ba-dead-455b-ac3f-78d65f54ef30-kube-api-access-jgm2h\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-hostroot\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-modprobe-d\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-registration-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ksvc\" (UniqueName: \"kubernetes.io/projected/20d23e7e-85a1-4f3f-80a9-4061a4109992-kube-api-access-6ksvc\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-slash\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7dc30f2a-cb78-4b51-876b-9f06d70cbcc6-iptables-alerter-script\") pod \"iptables-alerter-vrcx8\" (UID: \"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6\") " pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-cni-dir\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-run\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-cni-dir\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.783046 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780922 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-run\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-run-ovn-kubernetes\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-cnibin\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780964 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-registration-dir\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pd8m\" (UniqueName: \"kubernetes.io/projected/5fa014da-b395-4e6a-8804-5d3217305e99-kube-api-access-6pd8m\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.780991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/825150de-fdc4-4920-a483-b648d894dffc-hosts-file\") pod \"node-resolver-jv8n9\" (UID: \"825150de-fdc4-4920-a483-b648d894dffc\") " pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-var-lib-openvswitch\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781020 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-modprobe-d\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781083 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21f9b4ba-dead-455b-ac3f-78d65f54ef30-cni-binary-copy\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-socket-dir-parent\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.781108 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90688000-32fe-4a0b-8dc3-32e98e1fd32c-serviceca\") pod \"node-ca-h2rbz\" (UID: \"90688000-32fe-4a0b-8dc3-32e98e1fd32c\") " pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-slash\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.783710 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.781165 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret podName:bbc9f1d6-06c2-4142-bf58-7b25b80927d4 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:35.281149182 +0000 UTC m=+4.550632847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret") pod "global-pull-secret-syncer-c7ls5" (UID: "bbc9f1d6-06c2-4142-bf58-7b25b80927d4") : object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-sysctl-conf\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-sys\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-tmp\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/825150de-fdc4-4920-a483-b648d894dffc-hosts-file\") pod \"node-resolver-jv8n9\" (UID: \"825150de-fdc4-4920-a483-b648d894dffc\") " pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbc8\" (UniqueName: \"kubernetes.io/projected/6341cb8a-164f-4125-8aae-b7065e0d7b10-kube-api-access-kpbc8\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781269 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-sys\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781213 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-var-lib-openvswitch\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-system-cni-dir\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-var-lib-kubelet\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781329 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5fa014da-b395-4e6a-8804-5d3217305e99-cnibin\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rcl\" (UniqueName: \"kubernetes.io/projected/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-kube-api-access-t6rcl\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96dcb202-5818-45a8-a772-b148517779f2-agent-certs\") pod \"konnectivity-agent-fpdbr\" (UID: \"96dcb202-5818-45a8-a772-b148517779f2\") " pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-system-cni-dir\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-etc-selinux\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.784310 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8flg\" (UniqueName: \"kubernetes.io/projected/825150de-fdc4-4920-a483-b648d894dffc-kube-api-access-q8flg\") pod \"node-resolver-jv8n9\" (UID: \"825150de-fdc4-4920-a483-b648d894dffc\") " pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781497 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-sysctl-conf\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781508 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/90688000-32fe-4a0b-8dc3-32e98e1fd32c-serviceca\") pod \"node-ca-h2rbz\" (UID: \"90688000-32fe-4a0b-8dc3-32e98e1fd32c\") " pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781476 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6341cb8a-164f-4125-8aae-b7065e0d7b10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6341cb8a-164f-4125-8aae-b7065e0d7b10-ovnkube-script-lib\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-run-k8s-cni-cncf-io\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-var-lib-kubelet\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-multus-socket-dir-parent\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-etc-kubernetes\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-systemd\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-systemd\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781758 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-host-run-k8s-cni-cncf-io\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-cnibin\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21f9b4ba-dead-455b-ac3f-78d65f54ef30-etc-kubernetes\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.781884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/20d23e7e-85a1-4f3f-80a9-4061a4109992-etc-selinux\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.784951 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.782106 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6341cb8a-164f-4125-8aae-b7065e0d7b10-ovnkube-script-lib\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.785479 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.783927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-etc-tuned\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.785479 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.783962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6341cb8a-164f-4125-8aae-b7065e0d7b10-ovn-node-metrics-cert\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.785479 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.784139 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96dcb202-5818-45a8-a772-b148517779f2-agent-certs\") pod \"konnectivity-agent-fpdbr\" (UID: \"96dcb202-5818-45a8-a772-b148517779f2\") " pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:34.785479 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.784781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-tmp\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.788442 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.788421 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 04:59:34.788442 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.788443 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 04:59:34.788600 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.788456 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hb9c5 for pod openshift-network-diagnostics/network-check-target-7gpl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:34.788600 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:34.788509 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5 podName:d9d37572-4bb0-4aaf-8f04-14613b3719d1 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:35.2884936 +0000 UTC m=+4.557977283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hb9c5" (UniqueName: "kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5") pod "network-check-target-7gpl4" (UID: "d9d37572-4bb0-4aaf-8f04-14613b3719d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:34.789518 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.789451 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx86g\" (UniqueName: \"kubernetes.io/projected/7dc30f2a-cb78-4b51-876b-9f06d70cbcc6-kube-api-access-kx86g\") pod \"iptables-alerter-vrcx8\" (UID: \"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6\") " pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.791006 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.790959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rcl\" (UniqueName: \"kubernetes.io/projected/a32bb7b6-bb05-4e56-9e13-56e4c85e66bc-kube-api-access-t6rcl\") pod \"tuned-gwx25\" (UID: \"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc\") " pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.792334 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.792308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pd8m\" (UniqueName: \"kubernetes.io/projected/5fa014da-b395-4e6a-8804-5d3217305e99-kube-api-access-6pd8m\") pod \"multus-additional-cni-plugins-5gnfx\" (UID: \"5fa014da-b395-4e6a-8804-5d3217305e99\") " pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.792809 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.792785 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8flg\" (UniqueName: \"kubernetes.io/projected/825150de-fdc4-4920-a483-b648d894dffc-kube-api-access-q8flg\") pod \"node-resolver-jv8n9\" (UID: \"825150de-fdc4-4920-a483-b648d894dffc\") " pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.792896 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.792850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ksvc\" (UniqueName: \"kubernetes.io/projected/20d23e7e-85a1-4f3f-80a9-4061a4109992-kube-api-access-6ksvc\") pod \"aws-ebs-csi-driver-node-7j76v\" (UID: \"20d23e7e-85a1-4f3f-80a9-4061a4109992\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.792976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.792950 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2nh\" (UniqueName: \"kubernetes.io/projected/90688000-32fe-4a0b-8dc3-32e98e1fd32c-kube-api-access-8w2nh\") pod \"node-ca-h2rbz\" (UID: \"90688000-32fe-4a0b-8dc3-32e98e1fd32c\") " pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.793664 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.793640 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgm2h\" (UniqueName: \"kubernetes.io/projected/21f9b4ba-dead-455b-ac3f-78d65f54ef30-kube-api-access-jgm2h\") pod \"multus-vt4vk\" (UID: \"21f9b4ba-dead-455b-ac3f-78d65f54ef30\") " pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.794695 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.794678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnj5h\" (UniqueName: \"kubernetes.io/projected/8e5b3021-37f7-4153-bb80-758fb934948c-kube-api-access-rnj5h\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:34.794848 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.794823 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbc8\" (UniqueName: \"kubernetes.io/projected/6341cb8a-164f-4125-8aae-b7065e0d7b10-kube-api-access-kpbc8\") pod \"ovnkube-node-bnv7f\" (UID: \"6341cb8a-164f-4125-8aae-b7065e0d7b10\") " pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.892788 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.892706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" Jul 20 04:59:34.902373 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.902356 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jv8n9" Jul 20 04:59:34.907949 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.907933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h2rbz" Jul 20 04:59:34.913898 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.913879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gwx25" Jul 20 04:59:34.920754 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.920733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 04:59:34.927371 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.927353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" Jul 20 04:59:34.928426 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:34.928399 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825150de_fdc4_4920_a483_b648d894dffc.slice/crio-17281f687073fd9b07ca1525b1227d32a4c182a676eeb2e8e0e41d5784a024e5 WatchSource:0}: Error finding container 17281f687073fd9b07ca1525b1227d32a4c182a676eeb2e8e0e41d5784a024e5: Status 404 returned error can't find the container with id 17281f687073fd9b07ca1525b1227d32a4c182a676eeb2e8e0e41d5784a024e5 Jul 20 04:59:34.929310 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:34.929284 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90688000_32fe_4a0b_8dc3_32e98e1fd32c.slice/crio-f1654f1eb0103b973d773e29957da36b67b57d9f1a912d63b21144d1c74d785a WatchSource:0}: Error finding container f1654f1eb0103b973d773e29957da36b67b57d9f1a912d63b21144d1c74d785a: Status 404 returned error can't find the container with id f1654f1eb0103b973d773e29957da36b67b57d9f1a912d63b21144d1c74d785a Jul 20 04:59:34.933222 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.933182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vt4vk" Jul 20 04:59:34.939647 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.939627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:34.941906 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:34.941880 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa014da_b395_4e6a_8804_5d3217305e99.slice/crio-60b9ef97a37b496ee8f81725cd26a015bd27ed30ad60c7e2a102083ce8539bf6 WatchSource:0}: Error finding container 60b9ef97a37b496ee8f81725cd26a015bd27ed30ad60c7e2a102083ce8539bf6: Status 404 returned error can't find the container with id 60b9ef97a37b496ee8f81725cd26a015bd27ed30ad60c7e2a102083ce8539bf6 Jul 20 04:59:34.943144 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:34.943119 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6341cb8a_164f_4125_8aae_b7065e0d7b10.slice/crio-548eac6c3c0aebb3a995f029c79395a9d5846a601aae25acf3bff69560f0362c WatchSource:0}: Error finding container 548eac6c3c0aebb3a995f029c79395a9d5846a601aae25acf3bff69560f0362c: Status 404 returned error can't find the container with id 548eac6c3c0aebb3a995f029c79395a9d5846a601aae25acf3bff69560f0362c Jul 20 04:59:34.946029 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:34.946010 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f9b4ba_dead_455b_ac3f_78d65f54ef30.slice/crio-d9ddf47d71e910182bb74d49417e688692fc11912f6f2a993df72e510fa8b336 WatchSource:0}: Error finding container d9ddf47d71e910182bb74d49417e688692fc11912f6f2a993df72e510fa8b336: Status 404 returned error can't find the container with id d9ddf47d71e910182bb74d49417e688692fc11912f6f2a993df72e510fa8b336 Jul 20 04:59:34.947809 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:34.947788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vrcx8" Jul 20 04:59:34.951539 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:34.951516 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96dcb202_5818_45a8_a772_b148517779f2.slice/crio-bf7ea50541661fdacad1f4ba1f9ab5e353e8047ee06f579d89386505dc9cdab7 WatchSource:0}: Error finding container bf7ea50541661fdacad1f4ba1f9ab5e353e8047ee06f579d89386505dc9cdab7: Status 404 returned error can't find the container with id bf7ea50541661fdacad1f4ba1f9ab5e353e8047ee06f579d89386505dc9cdab7 Jul 20 04:59:34.962390 ip-10-0-133-196 kubenswrapper[2579]: W0720 04:59:34.962363 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc30f2a_cb78_4b51_876b_9f06d70cbcc6.slice/crio-4798f1f74bcbbb1713c61e457ca3f481ba40d0766a896205dbb23bd5ce0b858a WatchSource:0}: Error finding container 4798f1f74bcbbb1713c61e457ca3f481ba40d0766a896205dbb23bd5ce0b858a: Status 404 returned error can't find the container with id 4798f1f74bcbbb1713c61e457ca3f481ba40d0766a896205dbb23bd5ce0b858a Jul 20 04:59:35.286410 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.286142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:35.286567 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:35.286251 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:35.286567 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.286463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:35.286567 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:35.286515 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs podName:8e5b3021-37f7-4153-bb80-758fb934948c nodeName:}" failed. No retries permitted until 2026-07-20 04:59:36.286490748 +0000 UTC m=+5.555974426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs") pod "network-metrics-daemon-krtbd" (UID: "8e5b3021-37f7-4153-bb80-758fb934948c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:35.286567 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:35.286568 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:35.286752 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:35.286617 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret podName:bbc9f1d6-06c2-4142-bf58-7b25b80927d4 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:36.286601431 +0000 UTC m=+5.556085098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret") pod "global-pull-secret-syncer-c7ls5" (UID: "bbc9f1d6-06c2-4142-bf58-7b25b80927d4") : object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:35.387937 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.387252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:35.387937 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:35.387479 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 04:59:35.387937 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:35.387501 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 04:59:35.387937 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:35.387514 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hb9c5 for pod openshift-network-diagnostics/network-check-target-7gpl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:35.387937 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:35.387571 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5 podName:d9d37572-4bb0-4aaf-8f04-14613b3719d1 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:36.387552973 +0000 UTC m=+5.657036642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb9c5" (UniqueName: "kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5") pod "network-check-target-7gpl4" (UID: "d9d37572-4bb0-4aaf-8f04-14613b3719d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:35.581197 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.581114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal" event={"ID":"94f9424feb20d8f7b53d0fbc4a5441ee","Type":"ContainerStarted","Data":"34dbe7b941bd1d20739432f4f940a6bda0af8244a968bc0aaf2d1ed2f635ce9b"} Jul 20 04:59:35.595184 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.595124 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-196.ec2.internal" podStartSLOduration=2.595104619 podStartE2EDuration="2.595104619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:35.594870243 +0000 UTC m=+4.864354125" watchObservedRunningTime="2026-07-20 04:59:35.595104619 +0000 UTC m=+4.864588305" Jul 20 04:59:35.605654 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.605557 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 04:54:34 +0000 UTC" deadline="2028-03-30 06:12:11.561932144 +0000 UTC" Jul 20 04:59:35.605654 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.605609 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14857h12m35.95632727s" Jul 20 04:59:35.623888 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.623856 2579 generic.go:356] "Generic (PLEG): container finished" podID="dc92cae0202c6293dc0e89c9e7409eef" containerID="884434a58f5bb50cff89ab9a4806aa9628c566313a5af197a2ef7ef08d210a66" exitCode=0 Jul 20 04:59:35.624088 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.623967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" event={"ID":"dc92cae0202c6293dc0e89c9e7409eef","Type":"ContainerDied","Data":"884434a58f5bb50cff89ab9a4806aa9628c566313a5af197a2ef7ef08d210a66"} Jul 20 04:59:35.633359 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.633293 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vrcx8" event={"ID":"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6","Type":"ContainerStarted","Data":"4798f1f74bcbbb1713c61e457ca3f481ba40d0766a896205dbb23bd5ce0b858a"} Jul 20 04:59:35.638885 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.638806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fpdbr" event={"ID":"96dcb202-5818-45a8-a772-b148517779f2","Type":"ContainerStarted","Data":"bf7ea50541661fdacad1f4ba1f9ab5e353e8047ee06f579d89386505dc9cdab7"} Jul 20 04:59:35.641976 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.641908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" event={"ID":"5fa014da-b395-4e6a-8804-5d3217305e99","Type":"ContainerStarted","Data":"60b9ef97a37b496ee8f81725cd26a015bd27ed30ad60c7e2a102083ce8539bf6"} Jul 20 04:59:35.646267 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.646173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" event={"ID":"20d23e7e-85a1-4f3f-80a9-4061a4109992","Type":"ContainerStarted","Data":"ca8c0c17c10c6672ee4d7d24bf4dff28052ba341395df47dc0f29e8e88f867bd"} Jul 20 04:59:35.650829 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.650747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h2rbz" event={"ID":"90688000-32fe-4a0b-8dc3-32e98e1fd32c","Type":"ContainerStarted","Data":"f1654f1eb0103b973d773e29957da36b67b57d9f1a912d63b21144d1c74d785a"} Jul 20 04:59:35.654304 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.654243 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jv8n9" event={"ID":"825150de-fdc4-4920-a483-b648d894dffc","Type":"ContainerStarted","Data":"17281f687073fd9b07ca1525b1227d32a4c182a676eeb2e8e0e41d5784a024e5"} Jul 20 04:59:35.660851 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.660772 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vt4vk" event={"ID":"21f9b4ba-dead-455b-ac3f-78d65f54ef30","Type":"ContainerStarted","Data":"d9ddf47d71e910182bb74d49417e688692fc11912f6f2a993df72e510fa8b336"} Jul 20 04:59:35.670523 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.670484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" event={"ID":"6341cb8a-164f-4125-8aae-b7065e0d7b10","Type":"ContainerStarted","Data":"548eac6c3c0aebb3a995f029c79395a9d5846a601aae25acf3bff69560f0362c"} Jul 20 04:59:35.671933 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:35.671881 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gwx25" event={"ID":"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc","Type":"ContainerStarted","Data":"46426936ba53e98cdfe657cf03874298390a6d2050934e977bf5cf220c2c292c"} Jul 20 04:59:36.149128 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.149032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:36.149293 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.149183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:36.149647 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.149606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:36.149753 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.149718 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:36.149827 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.149786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:36.149878 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.149863 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:36.296376 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.296336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:36.296555 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.296434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:36.296636 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.296573 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:36.296691 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.296668 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret podName:bbc9f1d6-06c2-4142-bf58-7b25b80927d4 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:38.29664897 +0000 UTC m=+7.566132641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret") pod "global-pull-secret-syncer-c7ls5" (UID: "bbc9f1d6-06c2-4142-bf58-7b25b80927d4") : object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:36.297171 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.297119 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:36.297277 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.297179 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs podName:8e5b3021-37f7-4153-bb80-758fb934948c nodeName:}" failed. No retries permitted until 2026-07-20 04:59:38.297163285 +0000 UTC m=+7.566646949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs") pod "network-metrics-daemon-krtbd" (UID: "8e5b3021-37f7-4153-bb80-758fb934948c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:36.398309 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.397714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:36.398309 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.397862 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 04:59:36.398309 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.397881 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 04:59:36.398309 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.397893 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hb9c5 for pod openshift-network-diagnostics/network-check-target-7gpl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:36.398309 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:36.397956 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5 podName:d9d37572-4bb0-4aaf-8f04-14613b3719d1 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:38.397936554 +0000 UTC m=+7.667420219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb9c5" (UniqueName: "kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5") pod "network-check-target-7gpl4" (UID: "d9d37572-4bb0-4aaf-8f04-14613b3719d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:36.606249 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.606154 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 04:54:34 +0000 UTC" deadline="2028-03-17 06:13:04.432834082 +0000 UTC" Jul 20 04:59:36.606249 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.606199 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14545h13m27.826640353s" Jul 20 04:59:36.698910 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.698870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" event={"ID":"dc92cae0202c6293dc0e89c9e7409eef","Type":"ContainerStarted","Data":"6e71e2980a3d1876ddcc5876bc4a27332203ad2ca75d07ebc57669999b7a8b1a"} Jul 20 04:59:36.713911 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:36.713348 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-196.ec2.internal" podStartSLOduration=3.713326758 podStartE2EDuration="3.713326758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:36.712731035 +0000 UTC m=+5.982214721" watchObservedRunningTime="2026-07-20 04:59:36.713326758 +0000 UTC m=+5.982810446" Jul 20 04:59:38.149947 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:38.149913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:38.150429 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:38.149960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:38.150429 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.150098 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:38.150429 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.150129 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:38.150429 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:38.150161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:38.150429 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.150250 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:38.315363 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:38.314616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:38.315363 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:38.314684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:38.315363 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.314819 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:38.315363 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.314883 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret podName:bbc9f1d6-06c2-4142-bf58-7b25b80927d4 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:42.314863903 +0000 UTC m=+11.584347573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret") pod "global-pull-secret-syncer-c7ls5" (UID: "bbc9f1d6-06c2-4142-bf58-7b25b80927d4") : object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:38.315363 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.315277 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:38.315363 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.315324 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs podName:8e5b3021-37f7-4153-bb80-758fb934948c nodeName:}" failed. No retries permitted until 2026-07-20 04:59:42.315308819 +0000 UTC m=+11.584792505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs") pod "network-metrics-daemon-krtbd" (UID: "8e5b3021-37f7-4153-bb80-758fb934948c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:38.415444 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:38.415352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:38.415609 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.415549 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 04:59:38.415609 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.415570 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 04:59:38.415609 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.415582 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hb9c5 for pod openshift-network-diagnostics/network-check-target-7gpl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:38.415748 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:38.415641 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5 podName:d9d37572-4bb0-4aaf-8f04-14613b3719d1 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:42.41562168 +0000 UTC m=+11.685105359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb9c5" (UniqueName: "kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5") pod "network-check-target-7gpl4" (UID: "d9d37572-4bb0-4aaf-8f04-14613b3719d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:40.150332 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:40.149826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:40.150332 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:40.149867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:40.150332 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:40.149826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:40.150332 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:40.149961 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:40.150332 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:40.150072 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:40.150332 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:40.150147 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:42.149262 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:42.149224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:42.149887 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.149374 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:42.149887 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:42.149740 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:42.149887 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.149831 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:42.149887 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:42.149875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:42.150145 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.149947 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:42.348005 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:42.347965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:42.348206 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:42.348069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:42.348206 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.348185 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:42.348308 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.348251 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret podName:bbc9f1d6-06c2-4142-bf58-7b25b80927d4 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:50.34823141 +0000 UTC m=+19.617715154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret") pod "global-pull-secret-syncer-c7ls5" (UID: "bbc9f1d6-06c2-4142-bf58-7b25b80927d4") : object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:42.348308 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.348185 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:42.348401 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.348312 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs podName:8e5b3021-37f7-4153-bb80-758fb934948c nodeName:}" failed. No retries permitted until 2026-07-20 04:59:50.348300271 +0000 UTC m=+19.617783935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs") pod "network-metrics-daemon-krtbd" (UID: "8e5b3021-37f7-4153-bb80-758fb934948c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:42.448743 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:42.448555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:42.448921 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.448742 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 04:59:42.448921 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.448770 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 04:59:42.448921 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.448784 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hb9c5 for pod openshift-network-diagnostics/network-check-target-7gpl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:42.448921 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:42.448851 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5 podName:d9d37572-4bb0-4aaf-8f04-14613b3719d1 nodeName:}" failed. No retries permitted until 2026-07-20 04:59:50.44883201 +0000 UTC m=+19.718315676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb9c5" (UniqueName: "kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5") pod "network-check-target-7gpl4" (UID: "d9d37572-4bb0-4aaf-8f04-14613b3719d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:44.149842 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:44.149801 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:44.149842 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:44.149837 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:44.150364 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:44.149849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:44.150364 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:44.149923 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:44.150364 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:44.149970 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:44.150364 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:44.150048 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:46.150020 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:46.149979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:46.150549 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:46.150000 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:46.150549 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:46.150130 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:46.150549 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:46.150124 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:46.150549 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:46.150259 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:46.150549 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:46.150368 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:48.149487 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:48.149446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:48.149487 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:48.149486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:48.150017 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:48.149446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:48.150017 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:48.149593 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:48.150017 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:48.149628 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:48.150017 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:48.149693 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:50.149319 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:50.149287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:50.149806 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:50.149287 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:50.149806 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.149401 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:50.149806 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:50.149414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:50.149806 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.149524 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:50.149806 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.149612 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:50.409737 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:50.409645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:50.409737 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:50.409710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:50.410007 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.409820 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:50.410007 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.409848 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:50.410007 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.409897 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs podName:8e5b3021-37f7-4153-bb80-758fb934948c nodeName:}" failed. No retries permitted until 2026-07-20 05:00:06.409874916 +0000 UTC m=+35.679358582 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs") pod "network-metrics-daemon-krtbd" (UID: "8e5b3021-37f7-4153-bb80-758fb934948c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 04:59:50.410007 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.409925 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret podName:bbc9f1d6-06c2-4142-bf58-7b25b80927d4 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:06.409909413 +0000 UTC m=+35.679393078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret") pod "global-pull-secret-syncer-c7ls5" (UID: "bbc9f1d6-06c2-4142-bf58-7b25b80927d4") : object "kube-system"/"original-pull-secret" not registered Jul 20 04:59:50.510440 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:50.510401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:50.510635 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.510589 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 04:59:50.510635 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.510619 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 04:59:50.510635 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.510633 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hb9c5 for pod openshift-network-diagnostics/network-check-target-7gpl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:50.510793 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:50.510695 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5 podName:d9d37572-4bb0-4aaf-8f04-14613b3719d1 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:06.510676387 +0000 UTC m=+35.780160050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb9c5" (UniqueName: "kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5") pod "network-check-target-7gpl4" (UID: "d9d37572-4bb0-4aaf-8f04-14613b3719d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 04:59:52.149415 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:52.149376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:52.149415 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:52.149408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:52.149905 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:52.149388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:52.149905 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:52.149500 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:52.149905 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:52.149595 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:52.149905 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:52.149704 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:54.150116 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.149769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:54.150837 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.149800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:54.150837 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:54.150217 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:54.150837 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.149769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:54.150837 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:54.150260 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:54.150837 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:54.150366 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:54.741481 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.741442 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gwx25" event={"ID":"a32bb7b6-bb05-4e56-9e13-56e4c85e66bc","Type":"ContainerStarted","Data":"db7d140d31d4597295842cef9971920ef25423b929831b4cf34022ebdc955727"} Jul 20 04:59:54.742861 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.742838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fpdbr" event={"ID":"96dcb202-5818-45a8-a772-b148517779f2","Type":"ContainerStarted","Data":"b74aff461b375950d253e6e1c8d4f927e70c4ea7896f6c78e4bf59d13c5f678a"} Jul 20 04:59:54.744224 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.744201 2579 generic.go:356] "Generic (PLEG): container finished" podID="5fa014da-b395-4e6a-8804-5d3217305e99" containerID="11400e54a7302b2ee77b2123732468117f1c65d9d18643b900309429a3f27c5d" exitCode=0 Jul 20 04:59:54.744320 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.744266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" event={"ID":"5fa014da-b395-4e6a-8804-5d3217305e99","Type":"ContainerDied","Data":"11400e54a7302b2ee77b2123732468117f1c65d9d18643b900309429a3f27c5d"} Jul 20 04:59:54.745674 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.745627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" event={"ID":"20d23e7e-85a1-4f3f-80a9-4061a4109992","Type":"ContainerStarted","Data":"7e7be31592428a575dd1ad71b4cf7d0b4c2c615cb33a39310cf3c6c640428ff3"} Jul 20 04:59:54.747146 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.747124 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h2rbz" event={"ID":"90688000-32fe-4a0b-8dc3-32e98e1fd32c","Type":"ContainerStarted","Data":"bf24edf9ccac3facf9adb619a465c91d17e55bbfed8327b3502a088a0e0564f2"} Jul 20 04:59:54.748497 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.748478 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jv8n9" event={"ID":"825150de-fdc4-4920-a483-b648d894dffc","Type":"ContainerStarted","Data":"a1a9ff1fa36a79556988584c83a2d9abae96f2c6587892b4233045dbc69a9f18"} Jul 20 04:59:54.749885 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.749867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vt4vk" event={"ID":"21f9b4ba-dead-455b-ac3f-78d65f54ef30","Type":"ContainerStarted","Data":"ba4cf19e8cf0f47cebcf76aa99b1dba2f471075cdf2c434cb597878320d27164"} Jul 20 04:59:54.752631 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.752614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" event={"ID":"6341cb8a-164f-4125-8aae-b7065e0d7b10","Type":"ContainerStarted","Data":"de7044cfe5c4eb1d35bdeef917227f649af08e33fbf64bfd2096b3e8eedae92d"} Jul 20 04:59:54.752708 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.752663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" event={"ID":"6341cb8a-164f-4125-8aae-b7065e0d7b10","Type":"ContainerStarted","Data":"afb0c24af373518d0a9c57119d8721322a303e0d682031d4e551f0d22a954d24"} Jul 20 04:59:54.752708 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.752693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" event={"ID":"6341cb8a-164f-4125-8aae-b7065e0d7b10","Type":"ContainerStarted","Data":"793c32184f9314cd71da8a4d5210476ee80bac694ea0f4c0c9f2b619f3eb70e1"} Jul 20 04:59:54.752771 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.752722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" event={"ID":"6341cb8a-164f-4125-8aae-b7065e0d7b10","Type":"ContainerStarted","Data":"2c80acf4311798f71e8c7b20a0a7423ffab6916bba1039ffdfc10654b48512fb"} Jul 20 04:59:54.752771 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.752749 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" event={"ID":"6341cb8a-164f-4125-8aae-b7065e0d7b10","Type":"ContainerStarted","Data":"b89b559410dcc9dcc8f08e4bf65e142bc72dbe0b6e0ac8af320814d4cb63cbc7"} Jul 20 04:59:54.752834 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.752778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" event={"ID":"6341cb8a-164f-4125-8aae-b7065e0d7b10","Type":"ContainerStarted","Data":"b55af9ee77c58117712bc49d67637a49067425e99f8162f2f07472ea8c528948"} Jul 20 04:59:54.758109 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.758046 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gwx25" podStartSLOduration=5.06875071 podStartE2EDuration="23.758035253s" totalImagesPullingTime="18.689284543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:54.757389768 +0000 UTC m=+24.026873453" watchObservedRunningTime="2026-07-20 04:59:54.758035253 +0000 UTC m=+24.027518937" Jul 20 04:59:54.772914 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.772866 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fpdbr" podStartSLOduration=5.16558267 podStartE2EDuration="23.772854276s" totalImagesPullingTime="18.607271606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:54.772252553 +0000 UTC m=+24.041736237" watchObservedRunningTime="2026-07-20 04:59:54.772854276 +0000 UTC m=+24.042337960" Jul 20 04:59:54.804840 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.804793 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jv8n9" podStartSLOduration=5.110529189 podStartE2EDuration="23.804776849s" totalImagesPullingTime="18.69424766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:54.786111131 +0000 UTC m=+24.055594817" watchObservedRunningTime="2026-07-20 04:59:54.804776849 +0000 UTC m=+24.074260586" Jul 20 04:59:54.804996 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.804869 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h2rbz" podStartSLOduration=5.137366386 podStartE2EDuration="23.804863755s" totalImagesPullingTime="18.667497369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:54.80454896 +0000 UTC m=+24.074032646" watchObservedRunningTime="2026-07-20 04:59:54.804863755 +0000 UTC m=+24.074347492" Jul 20 04:59:54.841006 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:54.840955 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vt4vk" podStartSLOduration=5.040450062 podStartE2EDuration="23.840938609s" totalImagesPullingTime="18.800488547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:54.84025356 +0000 UTC m=+24.109737249" watchObservedRunningTime="2026-07-20 04:59:54.840938609 +0000 UTC m=+24.110422295" Jul 20 04:59:55.234940 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:55.234900 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 20 04:59:55.580626 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:55.580425 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T04:59:55.234931202Z","UUID":"e702bdc0-c58e-456c-9b91-c105e63042a8","Handler":null,"Name":"","Endpoint":""} Jul 20 04:59:55.582199 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:55.582172 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 20 04:59:55.582331 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:55.582221 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 20 04:59:55.757119 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:55.757083 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vrcx8" event={"ID":"7dc30f2a-cb78-4b51-876b-9f06d70cbcc6","Type":"ContainerStarted","Data":"446f8433bf80a67df1a4732c86dae7a7f416024ad7c931648a97d38ed519bc3f"} Jul 20 04:59:55.759260 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:55.759212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" event={"ID":"20d23e7e-85a1-4f3f-80a9-4061a4109992","Type":"ContainerStarted","Data":"15ff69e63442502ef82c792d5d4587e873846e07b84d5f0f1ecb8610c24777cd"} Jul 20 04:59:55.770012 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:55.769970 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vrcx8" podStartSLOduration=6.172683405 podStartE2EDuration="24.769956277s" totalImagesPullingTime="18.597272872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:55.769923975 +0000 UTC m=+25.039407660" watchObservedRunningTime="2026-07-20 04:59:55.769956277 +0000 UTC m=+25.039439961" Jul 20 04:59:56.149935 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:56.149903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:56.150131 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:56.149937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:56.150131 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:56.149937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:56.150131 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:56.150035 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:56.150303 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:56.150133 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:56.150303 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:56.150208 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:56.766324 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:56.766234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" event={"ID":"6341cb8a-164f-4125-8aae-b7065e0d7b10","Type":"ContainerStarted","Data":"21b02940b4b7f99af230ae6e1aa8920bdc4667f99db1112ef51c27782e6eeff6"} Jul 20 04:59:57.770960 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:57.770759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" event={"ID":"20d23e7e-85a1-4f3f-80a9-4061a4109992","Type":"ContainerStarted","Data":"701bdb05ee96e4d04d5acdeacacdc21ce75a95198be54e10bde68646595c94cd"} Jul 20 04:59:57.787914 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:57.787865 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7j76v" podStartSLOduration=5.319883792 podStartE2EDuration="26.787850932s" totalImagesPullingTime="21.46796714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:57.786687266 +0000 UTC m=+27.056170952" watchObservedRunningTime="2026-07-20 04:59:57.787850932 +0000 UTC m=+27.057334617" Jul 20 04:59:58.150137 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:58.150032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 04:59:58.150296 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:58.150032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 04:59:58.150296 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:58.150183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 04:59:58.150296 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:58.150032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 04:59:58.150296 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:58.150264 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 04:59:58.150527 ip-10-0-133-196 kubenswrapper[2579]: E0720 04:59:58.150354 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 04:59:58.502409 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:58.502326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:58.503193 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:58.503174 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:58.774137 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:58.774031 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:58.774560 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:58.774456 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fpdbr" Jul 20 04:59:59.777608 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:59.777367 2579 generic.go:356] "Generic (PLEG): container finished" podID="5fa014da-b395-4e6a-8804-5d3217305e99" containerID="d72f412c4923dd629a0f5b340a76addda58d854fabe536706b7f420c04774c74" exitCode=0 Jul 20 04:59:59.778528 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:59.777453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" event={"ID":"5fa014da-b395-4e6a-8804-5d3217305e99","Type":"ContainerDied","Data":"d72f412c4923dd629a0f5b340a76addda58d854fabe536706b7f420c04774c74"} Jul 20 04:59:59.781411 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:59.781388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" event={"ID":"6341cb8a-164f-4125-8aae-b7065e0d7b10","Type":"ContainerStarted","Data":"ed8e958a91fd8d0ada30cacb54e2679ca0dcd6cf2f2b8028a74ed4b0a91f52a0"} Jul 20 04:59:59.819701 ip-10-0-133-196 kubenswrapper[2579]: I0720 04:59:59.819644 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" podStartSLOduration=10.0354572 podStartE2EDuration="28.819629511s" totalImagesPullingTime="18.784172311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 04:59:59.817874175 +0000 UTC m=+29.087357859" watchObservedRunningTime="2026-07-20 04:59:59.819629511 +0000 UTC m=+29.089113196" Jul 20 05:00:00.149612 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:00.149493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 05:00:00.149612 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:00.149529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:00.149612 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:00.149585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:00:00.149612 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:00.149596 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 05:00:00.149879 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:00.149688 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 05:00:00.149879 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:00.149782 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 05:00:00.784444 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:00.784414 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 05:00:00.785212 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:00.784494 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 05:00:00.785212 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:00.784539 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 05:00:00.799533 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:00.799509 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 05:00:00.799676 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:00.799615 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 05:00:01.152640 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:01.152561 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c7ls5"] Jul 20 05:00:01.152640 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:01.152609 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7gpl4"] Jul 20 05:00:01.152852 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:01.152678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:01.152852 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:01.152766 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 05:00:01.152852 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:01.152791 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-krtbd"] Jul 20 05:00:01.152852 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:01.152817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 05:00:01.153008 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:01.152901 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 05:00:01.153008 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:01.152953 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:00:01.153125 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:01.153019 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 05:00:01.788363 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:01.788333 2579 generic.go:356] "Generic (PLEG): container finished" podID="5fa014da-b395-4e6a-8804-5d3217305e99" containerID="a4c57e360def75e37a63d500cc6f8ff2de0a3ecdd30c2d17cafa5fddc7c16da7" exitCode=0 Jul 20 05:00:01.788780 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:01.788418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" event={"ID":"5fa014da-b395-4e6a-8804-5d3217305e99","Type":"ContainerDied","Data":"a4c57e360def75e37a63d500cc6f8ff2de0a3ecdd30c2d17cafa5fddc7c16da7"} Jul 20 05:00:03.150144 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:03.149874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 05:00:03.150506 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:03.149887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:00:03.150506 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:03.150257 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 05:00:03.150506 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:03.149930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:03.150506 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:03.150296 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 05:00:03.150506 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:03.150415 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 05:00:03.795686 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:03.795651 2579 generic.go:356] "Generic (PLEG): container finished" podID="5fa014da-b395-4e6a-8804-5d3217305e99" containerID="5fe44ee8586e937cc28c4b18113f26d60f3b0b29b538100b76436b8fd047ed0b" exitCode=0 Jul 20 05:00:03.795846 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:03.795711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" event={"ID":"5fa014da-b395-4e6a-8804-5d3217305e99","Type":"ContainerDied","Data":"5fe44ee8586e937cc28c4b18113f26d60f3b0b29b538100b76436b8fd047ed0b"} Jul 20 05:00:05.149322 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:05.149286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:05.149966 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:05.149285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 05:00:05.149966 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:05.149431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:00:05.149966 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:05.149454 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7gpl4" podUID="d9d37572-4bb0-4aaf-8f04-14613b3719d1" Jul 20 05:00:05.149966 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:05.149562 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c7ls5" podUID="bbc9f1d6-06c2-4142-bf58-7b25b80927d4" Jul 20 05:00:05.149966 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:05.149664 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 05:00:06.429288 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.429249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 05:00:06.429713 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.429304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:00:06.429713 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.429399 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 05:00:06.429713 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.429407 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 20 05:00:06.429713 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.429457 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs podName:8e5b3021-37f7-4153-bb80-758fb934948c nodeName:}" failed. No retries permitted until 2026-07-20 05:00:38.429437981 +0000 UTC m=+67.698921654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs") pod "network-metrics-daemon-krtbd" (UID: "8e5b3021-37f7-4153-bb80-758fb934948c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 20 05:00:06.429713 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.429471 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret podName:bbc9f1d6-06c2-4142-bf58-7b25b80927d4 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:38.429464846 +0000 UTC m=+67.698948508 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret") pod "global-pull-secret-syncer-c7ls5" (UID: "bbc9f1d6-06c2-4142-bf58-7b25b80927d4") : object "kube-system"/"original-pull-secret" not registered Jul 20 05:00:06.514908 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.514825 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-196.ec2.internal" event="NodeReady" Jul 20 05:00:06.515046 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.514977 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 20 05:00:06.529848 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.529813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:06.530026 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.529983 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 20 05:00:06.530026 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.530004 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 20 05:00:06.530026 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.530014 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hb9c5 for pod openshift-network-diagnostics/network-check-target-7gpl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 05:00:06.530198 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.530089 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5 podName:d9d37572-4bb0-4aaf-8f04-14613b3719d1 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:38.530049385 +0000 UTC m=+67.799533051 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb9c5" (UniqueName: "kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5") pod "network-check-target-7gpl4" (UID: "d9d37572-4bb0-4aaf-8f04-14613b3719d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 20 05:00:06.558850 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.558818 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-95875"] Jul 20 05:00:06.562267 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.562246 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m59z4"] Jul 20 05:00:06.562401 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.562377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.564941 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.564915 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 20 05:00:06.565102 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.565040 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 20 05:00:06.565252 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.565238 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jlw9p\"" Jul 20 05:00:06.565675 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.565657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m59z4" Jul 20 05:00:06.567840 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.567804 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 20 05:00:06.567939 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.567875 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bbct5\"" Jul 20 05:00:06.567939 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.567895 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 20 05:00:06.568053 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.567875 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 20 05:00:06.574693 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.574671 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m59z4"] Jul 20 05:00:06.575378 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.575358 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-95875"] Jul 20 05:00:06.630218 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.630183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nk4m\" (UniqueName: \"kubernetes.io/projected/87705952-f6c4-4c65-83a7-1a04256c34b1-kube-api-access-9nk4m\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.630393 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.630232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ff43437-bb38-4b62-93e7-a9cd4cbc30d6-cert\") pod \"ingress-canary-m59z4\" (UID: \"7ff43437-bb38-4b62-93e7-a9cd4cbc30d6\") " pod="openshift-ingress-canary/ingress-canary-m59z4" Jul 20 05:00:06.630393 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.630268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.630393 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.630315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87705952-f6c4-4c65-83a7-1a04256c34b1-tmp-dir\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.630393 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.630370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/87705952-f6c4-4c65-83a7-1a04256c34b1-config-volume\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.630551 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.630393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb6wv\" (UniqueName: \"kubernetes.io/projected/7ff43437-bb38-4b62-93e7-a9cd4cbc30d6-kube-api-access-kb6wv\") pod \"ingress-canary-m59z4\" (UID: \"7ff43437-bb38-4b62-93e7-a9cd4cbc30d6\") " pod="openshift-ingress-canary/ingress-canary-m59z4" Jul 20 05:00:06.731175 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.731137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/87705952-f6c4-4c65-83a7-1a04256c34b1-config-volume\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.731175 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.731178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb6wv\" (UniqueName: \"kubernetes.io/projected/7ff43437-bb38-4b62-93e7-a9cd4cbc30d6-kube-api-access-kb6wv\") pod \"ingress-canary-m59z4\" (UID: \"7ff43437-bb38-4b62-93e7-a9cd4cbc30d6\") " pod="openshift-ingress-canary/ingress-canary-m59z4" Jul 20 05:00:06.731175 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.731222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nk4m\" (UniqueName: \"kubernetes.io/projected/87705952-f6c4-4c65-83a7-1a04256c34b1-kube-api-access-9nk4m\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.731547 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.731274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ff43437-bb38-4b62-93e7-a9cd4cbc30d6-cert\") pod \"ingress-canary-m59z4\" (UID: \"7ff43437-bb38-4b62-93e7-a9cd4cbc30d6\") " pod="openshift-ingress-canary/ingress-canary-m59z4" Jul 20 05:00:06.731547 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.731503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.731673 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.731563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87705952-f6c4-4c65-83a7-1a04256c34b1-tmp-dir\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.731727 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.731685 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 05:00:06.731778 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:06.731747 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls podName:87705952-f6c4-4c65-83a7-1a04256c34b1 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:07.231726821 +0000 UTC m=+36.501210489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls") pod "dns-default-95875" (UID: "87705952-f6c4-4c65-83a7-1a04256c34b1") : secret "dns-default-metrics-tls" not found Jul 20 05:00:06.731836 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.731803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/87705952-f6c4-4c65-83a7-1a04256c34b1-config-volume\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.731889 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.731864 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87705952-f6c4-4c65-83a7-1a04256c34b1-tmp-dir\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.736463 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.736438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ff43437-bb38-4b62-93e7-a9cd4cbc30d6-cert\") pod \"ingress-canary-m59z4\" (UID: \"7ff43437-bb38-4b62-93e7-a9cd4cbc30d6\") " pod="openshift-ingress-canary/ingress-canary-m59z4" Jul 20 05:00:06.740000 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.739949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb6wv\" (UniqueName: \"kubernetes.io/projected/7ff43437-bb38-4b62-93e7-a9cd4cbc30d6-kube-api-access-kb6wv\") pod \"ingress-canary-m59z4\" (UID: \"7ff43437-bb38-4b62-93e7-a9cd4cbc30d6\") " pod="openshift-ingress-canary/ingress-canary-m59z4" Jul 20 05:00:06.740418 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.740383 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nk4m\" (UniqueName: \"kubernetes.io/projected/87705952-f6c4-4c65-83a7-1a04256c34b1-kube-api-access-9nk4m\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:06.880091 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:06.880042 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m59z4" Jul 20 05:00:07.149554 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.149467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:00:07.149702 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.149467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:07.149702 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.149481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 05:00:07.152345 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.152314 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 05:00:07.152491 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.152348 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 05:00:07.152491 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.152445 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 05:00:07.153389 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.153368 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 05:00:07.153510 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.153368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z27sm\"" Jul 20 05:00:07.153510 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.153497 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-flt2r\"" Jul 20 05:00:07.233740 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:07.233697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:07.233902 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:07.233826 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 05:00:07.233902 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:07.233894 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls podName:87705952-f6c4-4c65-83a7-1a04256c34b1 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:08.233875157 +0000 UTC m=+37.503358824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls") pod "dns-default-95875" (UID: "87705952-f6c4-4c65-83a7-1a04256c34b1") : secret "dns-default-metrics-tls" not found Jul 20 05:00:08.242440 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:08.242401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:08.243211 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:08.242569 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 05:00:08.243211 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:08.242655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls podName:87705952-f6c4-4c65-83a7-1a04256c34b1 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:10.242633647 +0000 UTC m=+39.512117341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls") pod "dns-default-95875" (UID: "87705952-f6c4-4c65-83a7-1a04256c34b1") : secret "dns-default-metrics-tls" not found Jul 20 05:00:08.424651 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:08.424478 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m59z4"] Jul 20 05:00:08.427710 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:00:08.427686 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff43437_bb38_4b62_93e7_a9cd4cbc30d6.slice/crio-f49d35b7dbd15f6e68070aea4268e6c2598afc484ccc35466b265889e6993937 WatchSource:0}: Error finding container f49d35b7dbd15f6e68070aea4268e6c2598afc484ccc35466b265889e6993937: Status 404 returned error can't find the container with id f49d35b7dbd15f6e68070aea4268e6c2598afc484ccc35466b265889e6993937 Jul 20 05:00:08.807122 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:08.807020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m59z4" event={"ID":"7ff43437-bb38-4b62-93e7-a9cd4cbc30d6","Type":"ContainerStarted","Data":"f49d35b7dbd15f6e68070aea4268e6c2598afc484ccc35466b265889e6993937"} Jul 20 05:00:08.809676 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:08.809643 2579 generic.go:356] "Generic (PLEG): container finished" podID="5fa014da-b395-4e6a-8804-5d3217305e99" containerID="9270f8f7d3a1064d028187ee8e27896e6706e4accdbf21f973191c8bda946035" exitCode=0 Jul 20 05:00:08.809814 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:08.809686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" event={"ID":"5fa014da-b395-4e6a-8804-5d3217305e99","Type":"ContainerDied","Data":"9270f8f7d3a1064d028187ee8e27896e6706e4accdbf21f973191c8bda946035"} Jul 20 05:00:09.815644 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:09.815610 2579 generic.go:356] "Generic (PLEG): container finished" podID="5fa014da-b395-4e6a-8804-5d3217305e99" containerID="00bca067c59c868ea89b5e2fb9bd8dc3f780c5181bb8f77e7a0ed5a54cdf7713" exitCode=0 Jul 20 05:00:09.816161 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:09.815661 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" event={"ID":"5fa014da-b395-4e6a-8804-5d3217305e99","Type":"ContainerDied","Data":"00bca067c59c868ea89b5e2fb9bd8dc3f780c5181bb8f77e7a0ed5a54cdf7713"} Jul 20 05:00:10.260925 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:10.260825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:10.261088 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:10.261017 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 05:00:10.261141 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:10.261105 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls podName:87705952-f6c4-4c65-83a7-1a04256c34b1 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:14.261087769 +0000 UTC m=+43.530571433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls") pod "dns-default-95875" (UID: "87705952-f6c4-4c65-83a7-1a04256c34b1") : secret "dns-default-metrics-tls" not found Jul 20 05:00:10.822571 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:10.822537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" event={"ID":"5fa014da-b395-4e6a-8804-5d3217305e99","Type":"ContainerStarted","Data":"134f50e74dcb71656d3cbbf91fafa57eecd14778077ff6f5ea691e952d8f6f1e"} Jul 20 05:00:10.846976 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:10.846797 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5gnfx" podStartSLOduration=9.64561916 podStartE2EDuration="39.846780819s" totalImagesPullingTime="30.201161659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:00:10.845319611 +0000 UTC m=+40.114803296" watchObservedRunningTime="2026-07-20 05:00:10.846780819 +0000 UTC m=+40.116264503" Jul 20 05:00:11.826316 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:11.826271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m59z4" event={"ID":"7ff43437-bb38-4b62-93e7-a9cd4cbc30d6","Type":"ContainerStarted","Data":"b8fc205d7e0221705e16e3ef08b41a54b9fbebacf36342de3d39050b60f2b611"} Jul 20 05:00:11.840975 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:11.840925 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m59z4" podStartSLOduration=3.363162413 podStartE2EDuration="5.840909733s" totalImagesPullingTime="2.47774732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:00:11.84054277 +0000 UTC m=+41.110026456" watchObservedRunningTime="2026-07-20 05:00:11.840909733 +0000 UTC m=+41.110393417" Jul 20 05:00:14.288399 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:14.288355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:14.288766 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:14.288515 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 05:00:14.288766 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:14.288593 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls podName:87705952-f6c4-4c65-83a7-1a04256c34b1 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:22.288576958 +0000 UTC m=+51.558060621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls") pod "dns-default-95875" (UID: "87705952-f6c4-4c65-83a7-1a04256c34b1") : secret "dns-default-metrics-tls" not found Jul 20 05:00:22.341241 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:22.341194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:22.341707 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:22.341325 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 05:00:22.341707 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:22.341381 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls podName:87705952-f6c4-4c65-83a7-1a04256c34b1 nodeName:}" failed. No retries permitted until 2026-07-20 05:00:38.341364305 +0000 UTC m=+67.610847969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls") pod "dns-default-95875" (UID: "87705952-f6c4-4c65-83a7-1a04256c34b1") : secret "dns-default-metrics-tls" not found Jul 20 05:00:32.812566 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:32.812532 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bnv7f" Jul 20 05:00:38.358431 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.358389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:00:38.358896 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:38.358541 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 05:00:38.358896 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:38.358605 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls podName:87705952-f6c4-4c65-83a7-1a04256c34b1 nodeName:}" failed. No retries permitted until 2026-07-20 05:01:10.358587538 +0000 UTC m=+99.628071221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls") pod "dns-default-95875" (UID: "87705952-f6c4-4c65-83a7-1a04256c34b1") : secret "dns-default-metrics-tls" not found Jul 20 05:00:38.459519 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.459481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:00:38.459632 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.459534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 05:00:38.461958 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.461938 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 20 05:00:38.462001 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.461985 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 20 05:00:38.469845 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:38.469822 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 05:00:38.469939 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:00:38.469883 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs podName:8e5b3021-37f7-4153-bb80-758fb934948c nodeName:}" failed. No retries permitted until 2026-07-20 05:01:42.469866083 +0000 UTC m=+131.739349749 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs") pod "network-metrics-daemon-krtbd" (UID: "8e5b3021-37f7-4153-bb80-758fb934948c") : secret "metrics-daemon-secret" not found Jul 20 05:00:38.473492 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.473468 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbc9f1d6-06c2-4142-bf58-7b25b80927d4-original-pull-secret\") pod \"global-pull-secret-syncer-c7ls5\" (UID: \"bbc9f1d6-06c2-4142-bf58-7b25b80927d4\") " pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 05:00:38.560406 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.560352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:38.563356 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.563337 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 20 05:00:38.573727 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.573700 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 20 05:00:38.583878 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.583846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9c5\" (UniqueName: \"kubernetes.io/projected/d9d37572-4bb0-4aaf-8f04-14613b3719d1-kube-api-access-hb9c5\") pod \"network-check-target-7gpl4\" (UID: \"d9d37572-4bb0-4aaf-8f04-14613b3719d1\") " pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:38.671957 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.671855 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-flt2r\"" Jul 20 05:00:38.674796 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.674776 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c7ls5" Jul 20 05:00:38.680002 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.679985 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:38.803687 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.803658 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c7ls5"] Jul 20 05:00:38.809132 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:00:38.809047 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc9f1d6_06c2_4142_bf58_7b25b80927d4.slice/crio-36297b2f9f33689187793878db0d8c4d7d03a9331302bb923f94835b1d441568 WatchSource:0}: Error finding container 36297b2f9f33689187793878db0d8c4d7d03a9331302bb923f94835b1d441568: Status 404 returned error can't find the container with id 36297b2f9f33689187793878db0d8c4d7d03a9331302bb923f94835b1d441568 Jul 20 05:00:38.819896 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.819869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7gpl4"] Jul 20 05:00:38.823072 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:00:38.823020 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d37572_4bb0_4aaf_8f04_14613b3719d1.slice/crio-e86d74959da08fb2bacfb187cbbf15ada3b4c14089c00af5a40232edc759cacc WatchSource:0}: Error finding container e86d74959da08fb2bacfb187cbbf15ada3b4c14089c00af5a40232edc759cacc: Status 404 returned error can't find the container with id e86d74959da08fb2bacfb187cbbf15ada3b4c14089c00af5a40232edc759cacc Jul 20 05:00:38.889563 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.889529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7gpl4" event={"ID":"d9d37572-4bb0-4aaf-8f04-14613b3719d1","Type":"ContainerStarted","Data":"e86d74959da08fb2bacfb187cbbf15ada3b4c14089c00af5a40232edc759cacc"} Jul 20 05:00:38.890496 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:38.890475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c7ls5" event={"ID":"bbc9f1d6-06c2-4142-bf58-7b25b80927d4","Type":"ContainerStarted","Data":"36297b2f9f33689187793878db0d8c4d7d03a9331302bb923f94835b1d441568"} Jul 20 05:00:43.904826 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:43.904743 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7gpl4" event={"ID":"d9d37572-4bb0-4aaf-8f04-14613b3719d1","Type":"ContainerStarted","Data":"8044f6425fe9be74e258d41389cd15e7cae938ac8638dd28d4f3590d8bd08052"} Jul 20 05:00:43.905313 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:43.904874 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:00:43.906329 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:43.906307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c7ls5" event={"ID":"bbc9f1d6-06c2-4142-bf58-7b25b80927d4","Type":"ContainerStarted","Data":"15015abebec09f880cf10c4c036dda3dc0f8c3b536d5c4db3795429692b18e97"} Jul 20 05:00:43.919894 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:43.919848 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7gpl4" podStartSLOduration=68.107094321 podStartE2EDuration="1m12.919836269s" totalImagesPullingTime="4.812741948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:00:43.918309158 +0000 UTC m=+73.187792842" watchObservedRunningTime="2026-07-20 05:00:43.919836269 +0000 UTC m=+73.189319953" Jul 20 05:00:43.931313 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:00:43.931270 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c7ls5" podStartSLOduration=68.100173487 podStartE2EDuration="1m12.931256577s" totalImagesPullingTime="4.83108309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:00:43.931179189 +0000 UTC m=+73.200662879" watchObservedRunningTime="2026-07-20 05:00:43.931256577 +0000 UTC m=+73.200740262" Jul 20 05:01:10.379933 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:10.379870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:01:10.380454 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:01:10.380030 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 05:01:10.380454 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:01:10.380123 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls podName:87705952-f6c4-4c65-83a7-1a04256c34b1 nodeName:}" failed. No retries permitted until 2026-07-20 05:02:14.380107624 +0000 UTC m=+163.649591286 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls") pod "dns-default-95875" (UID: "87705952-f6c4-4c65-83a7-1a04256c34b1") : secret "dns-default-metrics-tls" not found Jul 20 05:01:12.545015 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.544977 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw"] Jul 20 05:01:12.549182 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.549163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" Jul 20 05:01:12.551794 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.551777 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 20 05:01:12.552926 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.552909 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 20 05:01:12.553015 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.552949 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 20 05:01:12.553015 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.552974 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 20 05:01:12.553015 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.552950 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-brx8l\"" Jul 20 05:01:12.554792 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.554773 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn"] Jul 20 05:01:12.557652 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.557630 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw"] Jul 20 05:01:12.557736 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.557724 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.560447 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.560424 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 20 05:01:12.560543 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.560508 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 20 05:01:12.560611 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.560581 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 20 05:01:12.560697 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.560678 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 20 05:01:12.569423 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.569401 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn"] Jul 20 05:01:12.695969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.695932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwgk\" (UniqueName: \"kubernetes.io/projected/53e35fb0-4c2d-47e9-a405-57db73feadce-kube-api-access-xqwgk\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.695969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.695971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-hub\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.696228 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.695991 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.696228 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.696080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.696228 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.696130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hzt\" (UniqueName: \"kubernetes.io/projected/311b377c-f835-4fc0-9903-208f472e0f9a-kube-api-access-p2hzt\") pod \"managed-serviceaccount-addon-agent-67db679bdb-nmmqw\" (UID: \"311b377c-f835-4fc0-9903-208f472e0f9a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" Jul 20 05:01:12.696228 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.696177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-ca\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.696228 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.696194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e35fb0-4c2d-47e9-a405-57db73feadce-additional-service-ca\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.696434 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.696282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/311b377c-f835-4fc0-9903-208f472e0f9a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67db679bdb-nmmqw\" (UID: \"311b377c-f835-4fc0-9903-208f472e0f9a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" Jul 20 05:01:12.796883 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.796790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwgk\" (UniqueName: \"kubernetes.io/projected/53e35fb0-4c2d-47e9-a405-57db73feadce-kube-api-access-xqwgk\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.796883 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.796836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-hub\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.796883 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.796860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.797191 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.796903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.797191 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.796936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hzt\" (UniqueName: \"kubernetes.io/projected/311b377c-f835-4fc0-9903-208f472e0f9a-kube-api-access-p2hzt\") pod \"managed-serviceaccount-addon-agent-67db679bdb-nmmqw\" (UID: \"311b377c-f835-4fc0-9903-208f472e0f9a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" Jul 20 05:01:12.797191 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.796985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-ca\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.797191 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.797012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e35fb0-4c2d-47e9-a405-57db73feadce-additional-service-ca\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.797394 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.797285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/311b377c-f835-4fc0-9903-208f472e0f9a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67db679bdb-nmmqw\" (UID: \"311b377c-f835-4fc0-9903-208f472e0f9a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" Jul 20 05:01:12.797854 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.797829 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/53e35fb0-4c2d-47e9-a405-57db73feadce-additional-service-ca\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.799569 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.799550 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.799862 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.799840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-ca\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.799941 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.799916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-hub\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.799990 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.799967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/53e35fb0-4c2d-47e9-a405-57db73feadce-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.800287 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.800267 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/311b377c-f835-4fc0-9903-208f472e0f9a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67db679bdb-nmmqw\" (UID: \"311b377c-f835-4fc0-9903-208f472e0f9a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" Jul 20 05:01:12.804589 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.804564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hzt\" (UniqueName: \"kubernetes.io/projected/311b377c-f835-4fc0-9903-208f472e0f9a-kube-api-access-p2hzt\") pod \"managed-serviceaccount-addon-agent-67db679bdb-nmmqw\" (UID: \"311b377c-f835-4fc0-9903-208f472e0f9a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" Jul 20 05:01:12.804813 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.804791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwgk\" (UniqueName: \"kubernetes.io/projected/53e35fb0-4c2d-47e9-a405-57db73feadce-kube-api-access-xqwgk\") pod \"cluster-proxy-proxy-agent-775dcbdc4-vtdrn\" (UID: \"53e35fb0-4c2d-47e9-a405-57db73feadce\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:12.875971 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.875932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" Jul 20 05:01:12.882142 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:12.882124 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:01:13.002496 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:13.002464 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw"] Jul 20 05:01:13.005898 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:01:13.005869 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod311b377c_f835_4fc0_9903_208f472e0f9a.slice/crio-676f77f17d4960996898e57b4c4d42ed0b09e4b542b94ee48df559450a960420 WatchSource:0}: Error finding container 676f77f17d4960996898e57b4c4d42ed0b09e4b542b94ee48df559450a960420: Status 404 returned error can't find the container with id 676f77f17d4960996898e57b4c4d42ed0b09e4b542b94ee48df559450a960420 Jul 20 05:01:13.022569 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:13.022545 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn"] Jul 20 05:01:13.025377 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:01:13.025355 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53e35fb0_4c2d_47e9_a405_57db73feadce.slice/crio-fa2627c5a71157360a51f9f2c5886487f1051fcbdd771b5d5ce2d81bdf91d5ef WatchSource:0}: Error finding container fa2627c5a71157360a51f9f2c5886487f1051fcbdd771b5d5ce2d81bdf91d5ef: Status 404 returned error can't find the container with id fa2627c5a71157360a51f9f2c5886487f1051fcbdd771b5d5ce2d81bdf91d5ef Jul 20 05:01:13.981567 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:13.981528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" event={"ID":"53e35fb0-4c2d-47e9-a405-57db73feadce","Type":"ContainerStarted","Data":"fa2627c5a71157360a51f9f2c5886487f1051fcbdd771b5d5ce2d81bdf91d5ef"} Jul 20 05:01:13.986181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:13.986145 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" event={"ID":"311b377c-f835-4fc0-9903-208f472e0f9a","Type":"ContainerStarted","Data":"676f77f17d4960996898e57b4c4d42ed0b09e4b542b94ee48df559450a960420"} Jul 20 05:01:14.911644 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:14.911599 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7gpl4" Jul 20 05:01:17.999842 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:17.999801 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" event={"ID":"311b377c-f835-4fc0-9903-208f472e0f9a","Type":"ContainerStarted","Data":"1a4c0cb151e31e5c2f8d4793e020e2a54d45fb6f79c9e0f38357cdb99c4a9142"} Jul 20 05:01:18.001830 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:18.001806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" event={"ID":"53e35fb0-4c2d-47e9-a405-57db73feadce","Type":"ContainerStarted","Data":"f3f23fd4733389bf32ec405497c90c258fcd6363925ed15391ff5192f0114bea"} Jul 20 05:01:18.001940 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:18.001837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" event={"ID":"53e35fb0-4c2d-47e9-a405-57db73feadce","Type":"ContainerStarted","Data":"38ab843ac38addfc4e9b40a589a3c16bf214e47dbacbcd732f0562f6234085c8"} Jul 20 05:01:18.001940 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:18.001849 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" event={"ID":"53e35fb0-4c2d-47e9-a405-57db73feadce","Type":"ContainerStarted","Data":"f79bd5f5c5fc7ae679d231f9892bdadbd79fe18e60442c0d29dedfdd200770f3"} Jul 20 05:01:18.013793 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:18.013745 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" podStartSLOduration=1.512513099 podStartE2EDuration="6.013730174s" totalImagesPullingTime="4.501217075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:01:18.012980245 +0000 UTC m=+107.282463931" watchObservedRunningTime="2026-07-20 05:01:18.013730174 +0000 UTC m=+107.283213858" Jul 20 05:01:18.029306 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:18.029260 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" podStartSLOduration=1.531759498 podStartE2EDuration="6.02924669s" totalImagesPullingTime="4.497487192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:01:18.028909684 +0000 UTC m=+107.298393370" watchObservedRunningTime="2026-07-20 05:01:18.02924669 +0000 UTC m=+107.298730376" Jul 20 05:01:42.512529 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:01:42.512482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:01:42.513039 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:01:42.512606 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 20 05:01:42.513039 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:01:42.512672 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs podName:8e5b3021-37f7-4153-bb80-758fb934948c nodeName:}" failed. No retries permitted until 2026-07-20 05:03:44.512656457 +0000 UTC m=+253.782140120 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs") pod "network-metrics-daemon-krtbd" (UID: "8e5b3021-37f7-4153-bb80-758fb934948c") : secret "metrics-daemon-secret" not found Jul 20 05:02:09.574693 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:02:09.574641 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-95875" podUID="87705952-f6c4-4c65-83a7-1a04256c34b1" Jul 20 05:02:10.153898 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:02:10.153866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95875" Jul 20 05:02:10.162800 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:02:10.162773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-krtbd" podUID="8e5b3021-37f7-4153-bb80-758fb934948c" Jul 20 05:02:14.431669 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:02:14.431630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:02:14.432093 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:02:14.431744 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 20 05:02:14.432093 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:02:14.431796 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls podName:87705952-f6c4-4c65-83a7-1a04256c34b1 nodeName:}" failed. No retries permitted until 2026-07-20 05:04:16.431781932 +0000 UTC m=+285.701265594 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls") pod "dns-default-95875" (UID: "87705952-f6c4-4c65-83a7-1a04256c34b1") : secret "dns-default-metrics-tls" not found Jul 20 05:02:23.150120 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:02:23.150041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:03:18.339519 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:18.339484 2579 generic.go:356] "Generic (PLEG): container finished" podID="311b377c-f835-4fc0-9903-208f472e0f9a" containerID="1a4c0cb151e31e5c2f8d4793e020e2a54d45fb6f79c9e0f38357cdb99c4a9142" exitCode=255 Jul 20 05:03:18.339519 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:18.339523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" event={"ID":"311b377c-f835-4fc0-9903-208f472e0f9a","Type":"ContainerDied","Data":"1a4c0cb151e31e5c2f8d4793e020e2a54d45fb6f79c9e0f38357cdb99c4a9142"} Jul 20 05:03:18.339975 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:18.339813 2579 scope.go:117] "RemoveContainer" containerID="1a4c0cb151e31e5c2f8d4793e020e2a54d45fb6f79c9e0f38357cdb99c4a9142" Jul 20 05:03:19.343977 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:19.343940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67db679bdb-nmmqw" event={"ID":"311b377c-f835-4fc0-9903-208f472e0f9a","Type":"ContainerStarted","Data":"4ec902d66fc405342fb9cc92132a4314c383442578da4051d3f1e23a270cd75e"} Jul 20 05:03:30.381805 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.381770 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sxxv5"] Jul 20 05:03:30.384768 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.384747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.387562 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.387524 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 20 05:03:30.387562 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.387535 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 20 05:03:30.387722 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.387562 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 20 05:03:30.387722 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.387611 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-28fj9\"" Jul 20 05:03:30.388638 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.388619 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 20 05:03:30.396633 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.396612 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxxv5"] Jul 20 05:03:30.477109 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.477048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.477297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.477139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-data-volume\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.477297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.477173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-crio-socket\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.477297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.477204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddx98\" (UniqueName: \"kubernetes.io/projected/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-kube-api-access-ddx98\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.477297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.477245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.577557 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.577511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.577750 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.577587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-data-volume\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.577750 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.577618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-crio-socket\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.577750 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:30.577628 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 20 05:03:30.577750 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.577648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddx98\" (UniqueName: \"kubernetes.io/projected/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-kube-api-access-ddx98\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.577750 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.577692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.577750 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:30.577709 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-kube-rbac-proxy-cm podName:6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01 nodeName:}" failed. No retries permitted until 2026-07-20 05:03:31.077692891 +0000 UTC m=+240.347176553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxxv5" (UID: "6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01") : configmap "kube-rbac-proxy" not found Jul 20 05:03:30.577750 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.577727 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-crio-socket\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.578031 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:30.577796 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 05:03:30.578031 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:30.577867 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls podName:6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01 nodeName:}" failed. No retries permitted until 2026-07-20 05:03:31.077850252 +0000 UTC m=+240.347333915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxxv5" (UID: "6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01") : secret "insights-runtime-extractor-tls" not found Jul 20 05:03:30.578031 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.577930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-data-volume\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:30.588106 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:30.588082 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddx98\" (UniqueName: \"kubernetes.io/projected/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-kube-api-access-ddx98\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:31.081300 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:31.081266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:31.081300 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:31.081303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:31.081498 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:31.081402 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 05:03:31.081498 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:31.081472 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls podName:6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01 nodeName:}" failed. No retries permitted until 2026-07-20 05:03:32.081456391 +0000 UTC m=+241.350940054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxxv5" (UID: "6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01") : secret "insights-runtime-extractor-tls" not found Jul 20 05:03:31.081775 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:31.081758 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:32.088837 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:32.088799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:32.089279 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:32.088931 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 05:03:32.089279 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:32.088992 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls podName:6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01 nodeName:}" failed. No retries permitted until 2026-07-20 05:03:34.088978061 +0000 UTC m=+243.358461724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxxv5" (UID: "6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01") : secret "insights-runtime-extractor-tls" not found Jul 20 05:03:33.364963 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:33.364934 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jv8n9_825150de-fdc4-4920-a483-b648d894dffc/dns-node-resolver/0.log" Jul 20 05:03:33.966457 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:33.966428 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h2rbz_90688000-32fe-4a0b-8dc3-32e98e1fd32c/node-ca/0.log" Jul 20 05:03:34.104669 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:34.104626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:34.104877 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:34.104814 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 05:03:34.104944 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:34.104898 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls podName:6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01 nodeName:}" failed. No retries permitted until 2026-07-20 05:03:38.104876364 +0000 UTC m=+247.374360029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxxv5" (UID: "6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01") : secret "insights-runtime-extractor-tls" not found Jul 20 05:03:34.767161 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:34.767129 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m59z4_7ff43437-bb38-4b62-93e7-a9cd4cbc30d6/serve-healthcheck-canary/0.log" Jul 20 05:03:38.135562 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:38.135510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:38.136015 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:38.135664 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 20 05:03:38.136015 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:03:38.135726 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls podName:6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01 nodeName:}" failed. No retries permitted until 2026-07-20 05:03:46.135710429 +0000 UTC m=+255.405194091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxxv5" (UID: "6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01") : secret "insights-runtime-extractor-tls" not found Jul 20 05:03:44.583898 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:44.583858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:03:44.586158 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:44.586138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e5b3021-37f7-4153-bb80-758fb934948c-metrics-certs\") pod \"network-metrics-daemon-krtbd\" (UID: \"8e5b3021-37f7-4153-bb80-758fb934948c\") " pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:03:44.753150 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:44.753117 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z27sm\"" Jul 20 05:03:44.761017 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:44.760996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-krtbd" Jul 20 05:03:44.884518 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:44.884481 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-krtbd"] Jul 20 05:03:44.887001 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:03:44.886976 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5b3021_37f7_4153_bb80_758fb934948c.slice/crio-ed027c2aa046ab67659f5e4c58cf7dc0567ec1d289cf88e63a71344f73912c14 WatchSource:0}: Error finding container ed027c2aa046ab67659f5e4c58cf7dc0567ec1d289cf88e63a71344f73912c14: Status 404 returned error can't find the container with id ed027c2aa046ab67659f5e4c58cf7dc0567ec1d289cf88e63a71344f73912c14 Jul 20 05:03:45.420163 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:45.420125 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krtbd" event={"ID":"8e5b3021-37f7-4153-bb80-758fb934948c","Type":"ContainerStarted","Data":"ed027c2aa046ab67659f5e4c58cf7dc0567ec1d289cf88e63a71344f73912c14"} Jul 20 05:03:46.195931 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:46.195903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:46.198251 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:46.198232 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxxv5\" (UID: \"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01\") " pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:46.295754 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:46.295716 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-28fj9\"" Jul 20 05:03:46.303897 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:46.303869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxxv5" Jul 20 05:03:46.424515 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:46.424484 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxxv5"] Jul 20 05:03:46.425995 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:46.425972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krtbd" event={"ID":"8e5b3021-37f7-4153-bb80-758fb934948c","Type":"ContainerStarted","Data":"8ae05256ce71230006fa5b70fbb0609411d5c205f0c3b340fa9b10fd44c334fc"} Jul 20 05:03:46.426111 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:46.426022 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-krtbd" event={"ID":"8e5b3021-37f7-4153-bb80-758fb934948c","Type":"ContainerStarted","Data":"c284d83e708e76d04dc55178e87c4a0aeef8aa9a6a6244424ccfaf96b6597314"} Jul 20 05:03:46.427716 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:03:46.427694 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdcdf91_bb28_42b4_9d0d_d8eb6ed02c01.slice/crio-d4a79bfae724a9ad3354b40549191715ccbde3ec02a286e418cbabace9f25dfc WatchSource:0}: Error finding container d4a79bfae724a9ad3354b40549191715ccbde3ec02a286e418cbabace9f25dfc: Status 404 returned error can't find the container with id d4a79bfae724a9ad3354b40549191715ccbde3ec02a286e418cbabace9f25dfc Jul 20 05:03:46.440295 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:46.440228 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-krtbd" podStartSLOduration=254.269585705 podStartE2EDuration="4m15.440212163s" totalImagesPullingTime="1.170626458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 04:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:03:46.439235647 +0000 UTC m=+255.708719331" watchObservedRunningTime="2026-07-20 05:03:46.440212163 +0000 UTC m=+255.709695848" Jul 20 05:03:47.430466 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:47.430425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxxv5" event={"ID":"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01","Type":"ContainerStarted","Data":"d4a79bfae724a9ad3354b40549191715ccbde3ec02a286e418cbabace9f25dfc"} Jul 20 05:03:51.444735 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:51.444695 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxxv5" event={"ID":"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01","Type":"ContainerStarted","Data":"91ace532ad2619f414dda5177a1775e9a41003551c2469797856b53749fa1233"} Jul 20 05:03:55.461275 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:55.461231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxxv5" event={"ID":"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01","Type":"ContainerStarted","Data":"d3ca472337f54e598cc546f6d715aee86ff988fad403d38c6a7252349f8999ac"} Jul 20 05:03:58.473834 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:58.473750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxxv5" event={"ID":"6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01","Type":"ContainerStarted","Data":"ee08d84c055e44d4f14f49eafa7be929de6b3b5cc47f9dd42b75b56be43dd19c"} Jul 20 05:03:58.490321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:03:58.490269 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sxxv5" podStartSLOduration=16.80893459 podStartE2EDuration="28.490254836s" totalImagesPullingTime="11.681320246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 05:03:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:03:58.48903163 +0000 UTC m=+267.758515316" watchObservedRunningTime="2026-07-20 05:03:58.490254836 +0000 UTC m=+267.759738521" Jul 20 05:04:00.108807 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.108769 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dc58cd79c-skjdb"] Jul 20 05:04:00.111704 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.111687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.114331 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.114304 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 05:04:00.114441 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.114343 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 05:04:00.115481 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.115461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 20 05:04:00.115587 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.115503 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-25rk9\"" Jul 20 05:04:00.115587 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.115508 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 20 05:04:00.115700 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.115591 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 20 05:04:00.115700 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.115594 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 20 05:04:00.115700 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.115634 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 20 05:04:00.126820 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.126795 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc58cd79c-skjdb"] Jul 20 05:04:00.209156 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.209120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5d5c\" (UniqueName: \"kubernetes.io/projected/3373bdb1-ea18-42e3-a929-485044a77fe6-kube-api-access-j5d5c\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.209156 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.209162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-oauth-config\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.209368 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.209238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-serving-cert\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.209368 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.209338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-service-ca\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.209368 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.209356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-oauth-serving-cert\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.209464 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.209374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-console-config\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.310046 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.310006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-service-ca\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.310046 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.310043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-oauth-serving-cert\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.310273 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.310098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-console-config\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.310273 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.310126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5d5c\" (UniqueName: \"kubernetes.io/projected/3373bdb1-ea18-42e3-a929-485044a77fe6-kube-api-access-j5d5c\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.310273 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.310257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-oauth-config\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.310431 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.310341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-serving-cert\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.310934 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.310908 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-service-ca\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.311072 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.310910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-oauth-serving-cert\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.311072 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.310910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-console-config\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.312627 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.312591 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-oauth-config\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.312760 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.312743 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-serving-cert\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.318289 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.318266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5d5c\" (UniqueName: \"kubernetes.io/projected/3373bdb1-ea18-42e3-a929-485044a77fe6-kube-api-access-j5d5c\") pod \"console-7dc58cd79c-skjdb\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.421084 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.420984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:00.544861 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:00.544827 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc58cd79c-skjdb"] Jul 20 05:04:00.548090 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:04:00.548045 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3373bdb1_ea18_42e3_a929_485044a77fe6.slice/crio-4505b98be56eaf447a89913ab23a6b0c0083d977fed87520a28a8d714dc381ea WatchSource:0}: Error finding container 4505b98be56eaf447a89913ab23a6b0c0083d977fed87520a28a8d714dc381ea: Status 404 returned error can't find the container with id 4505b98be56eaf447a89913ab23a6b0c0083d977fed87520a28a8d714dc381ea Jul 20 05:04:01.383680 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.383645 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828"] Jul 20 05:04:01.386737 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.386715 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" Jul 20 05:04:01.389262 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.389238 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 20 05:04:01.389376 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.389243 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qdsdc\"" Jul 20 05:04:01.398828 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.398801 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828"] Jul 20 05:04:01.485039 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.484993 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc58cd79c-skjdb" event={"ID":"3373bdb1-ea18-42e3-a929-485044a77fe6","Type":"ContainerStarted","Data":"4505b98be56eaf447a89913ab23a6b0c0083d977fed87520a28a8d714dc381ea"} Jul 20 05:04:01.519417 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.519381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8e17150b-521d-4acd-b809-7fad7c504398-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6t828\" (UID: \"8e17150b-521d-4acd-b809-7fad7c504398\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" Jul 20 05:04:01.620421 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.620376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8e17150b-521d-4acd-b809-7fad7c504398-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6t828\" (UID: \"8e17150b-521d-4acd-b809-7fad7c504398\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" Jul 20 05:04:01.623071 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.623030 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8e17150b-521d-4acd-b809-7fad7c504398-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6t828\" (UID: \"8e17150b-521d-4acd-b809-7fad7c504398\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" Jul 20 05:04:01.696720 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.696627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" Jul 20 05:04:01.841851 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:01.841816 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828"] Jul 20 05:04:01.845699 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:04:01.845667 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e17150b_521d_4acd_b809_7fad7c504398.slice/crio-1b0abba35c0f67caa8d54646857530eaf193f00c5b8dbd139d81b987247ee8eb WatchSource:0}: Error finding container 1b0abba35c0f67caa8d54646857530eaf193f00c5b8dbd139d81b987247ee8eb: Status 404 returned error can't find the container with id 1b0abba35c0f67caa8d54646857530eaf193f00c5b8dbd139d81b987247ee8eb Jul 20 05:04:02.489923 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:02.489882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" event={"ID":"8e17150b-521d-4acd-b809-7fad7c504398","Type":"ContainerStarted","Data":"1b0abba35c0f67caa8d54646857530eaf193f00c5b8dbd139d81b987247ee8eb"} Jul 20 05:04:03.496695 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:03.496653 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc58cd79c-skjdb" event={"ID":"3373bdb1-ea18-42e3-a929-485044a77fe6","Type":"ContainerStarted","Data":"9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495"} Jul 20 05:04:03.512877 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:03.512767 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dc58cd79c-skjdb" podStartSLOduration=0.807833847 podStartE2EDuration="3.512749637s" totalImagesPullingTime="2.70491579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:04:03.512237586 +0000 UTC m=+272.781721272" watchObservedRunningTime="2026-07-20 05:04:03.512749637 +0000 UTC m=+272.782233322" Jul 20 05:04:04.501597 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:04.501562 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" event={"ID":"8e17150b-521d-4acd-b809-7fad7c504398","Type":"ContainerStarted","Data":"6ac06f2e6383e354c4a58a530d13745ab93aecf311011f9e3676659912b3e3b1"} Jul 20 05:04:04.516274 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:04.516226 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" podStartSLOduration=1.381929381 podStartE2EDuration="3.516211226s" totalImagesPullingTime="2.134281845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:04:04.515861792 +0000 UTC m=+273.785345489" watchObservedRunningTime="2026-07-20 05:04:04.516211226 +0000 UTC m=+273.785694912" Jul 20 05:04:05.504657 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:05.504626 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" Jul 20 05:04:05.509321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:05.509296 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6t828" Jul 20 05:04:10.421451 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:10.421404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:10.422090 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:10.421480 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:10.426146 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:10.426123 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:10.525122 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:10.525092 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:11.820973 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.820932 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gh5fb"] Jul 20 05:04:11.828541 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.828513 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:11.831280 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.831239 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 20 05:04:11.831280 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.831270 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 20 05:04:11.831482 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.831395 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 20 05:04:11.832202 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.832179 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wkpjg\"" Jul 20 05:04:11.832311 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.832201 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 20 05:04:11.832356 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.832323 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 20 05:04:11.832420 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.832405 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 20 05:04:11.898825 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.898785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-textfile\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:11.898825 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.898829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b04d0856-7c69-4200-bf79-ac7bab49f604-sys\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:11.899086 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.898851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:11.899086 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.898957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b04d0856-7c69-4200-bf79-ac7bab49f604-root\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:11.899086 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.898992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-tls\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:11.899086 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.899016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqkk\" (UniqueName: \"kubernetes.io/projected/b04d0856-7c69-4200-bf79-ac7bab49f604-kube-api-access-pnqkk\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:11.899086 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.899081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04d0856-7c69-4200-bf79-ac7bab49f604-metrics-client-ca\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:11.899259 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.899154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-accelerators-collector-config\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:11.899259 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:11.899177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-wtmp\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-textfile\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b04d0856-7c69-4200-bf79-ac7bab49f604-sys\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b04d0856-7c69-4200-bf79-ac7bab49f604-root\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-tls\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqkk\" (UniqueName: \"kubernetes.io/projected/b04d0856-7c69-4200-bf79-ac7bab49f604-kube-api-access-pnqkk\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b04d0856-7c69-4200-bf79-ac7bab49f604-sys\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04d0856-7c69-4200-bf79-ac7bab49f604-metrics-client-ca\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000515 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b04d0856-7c69-4200-bf79-ac7bab49f604-root\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-accelerators-collector-config\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:04:12.000569 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 20 05:04:12.000615 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-wtmp\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.001012 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:04:12.000625 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-tls podName:b04d0856-7c69-4200-bf79-ac7bab49f604 nodeName:}" failed. No retries permitted until 2026-07-20 05:04:12.500607997 +0000 UTC m=+281.770091682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-tls") pod "node-exporter-gh5fb" (UID: "b04d0856-7c69-4200-bf79-ac7bab49f604") : secret "node-exporter-tls" not found Jul 20 05:04:12.001012 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000699 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-textfile\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.001012 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.000740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-wtmp\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.001166 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.001042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04d0856-7c69-4200-bf79-ac7bab49f604-metrics-client-ca\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.001166 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.001138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-accelerators-collector-config\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.002792 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.002775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.009983 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.009962 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqkk\" (UniqueName: \"kubernetes.io/projected/b04d0856-7c69-4200-bf79-ac7bab49f604-kube-api-access-pnqkk\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.504188 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.504143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-tls\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.506465 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.506435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b04d0856-7c69-4200-bf79-ac7bab49f604-node-exporter-tls\") pod \"node-exporter-gh5fb\" (UID: \"b04d0856-7c69-4200-bf79-ac7bab49f604\") " pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.629235 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.629204 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc58cd79c-skjdb"] Jul 20 05:04:12.738290 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.738232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gh5fb" Jul 20 05:04:12.750967 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:04:12.750942 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04d0856_7c69_4200_bf79_ac7bab49f604.slice/crio-9214b003ea312360a2cbf23f796f1f35b1c1a62133b099d39a3cd5f82df5aa0c WatchSource:0}: Error finding container 9214b003ea312360a2cbf23f796f1f35b1c1a62133b099d39a3cd5f82df5aa0c: Status 404 returned error can't find the container with id 9214b003ea312360a2cbf23f796f1f35b1c1a62133b099d39a3cd5f82df5aa0c Jul 20 05:04:12.861568 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.861531 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 20 05:04:12.866685 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.866668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:12.868885 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.868862 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 20 05:04:12.869285 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.869265 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 20 05:04:12.869520 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.869504 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 20 05:04:12.869520 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.869512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 20 05:04:12.869647 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.869512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rfbhb\"" Jul 20 05:04:12.869894 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.869875 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 20 05:04:12.869947 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.869917 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 20 05:04:12.870014 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.869950 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 20 05:04:12.870213 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.870198 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 20 05:04:12.870389 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.870375 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 20 05:04:12.882159 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.882134 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 20 05:04:12.883228 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:12.883195 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" podUID="53e35fb0-4c2d-47e9-a405-57db73feadce" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 05:04:13.006629 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-config-volume\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006629 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006629 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006629 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3278cd-1342-427d-80a2-25d62c461f3b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006682 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-web-config\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006811 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8555k\" (UniqueName: \"kubernetes.io/projected/ac3278cd-1342-427d-80a2-25d62c461f3b-kube-api-access-8555k\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac3278cd-1342-427d-80a2-25d62c461f3b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac3278cd-1342-427d-80a2-25d62c461f3b-config-out\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac3278cd-1342-427d-80a2-25d62c461f3b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.006916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.007221 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.006965 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac3278cd-1342-427d-80a2-25d62c461f3b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108152 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-config-volume\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108152 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108397 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108397 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3278cd-1342-427d-80a2-25d62c461f3b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108397 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108397 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-web-config\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108397 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108646 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8555k\" (UniqueName: \"kubernetes.io/projected/ac3278cd-1342-427d-80a2-25d62c461f3b-kube-api-access-8555k\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108646 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac3278cd-1342-427d-80a2-25d62c461f3b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108646 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac3278cd-1342-427d-80a2-25d62c461f3b-config-out\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108646 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac3278cd-1342-427d-80a2-25d62c461f3b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108646 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.108646 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac3278cd-1342-427d-80a2-25d62c461f3b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.109215 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.108991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac3278cd-1342-427d-80a2-25d62c461f3b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.109518 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.109455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac3278cd-1342-427d-80a2-25d62c461f3b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.110152 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.110105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac3278cd-1342-427d-80a2-25d62c461f3b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.111482 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.111455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-config-volume\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.111641 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.111577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.112218 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.111922 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac3278cd-1342-427d-80a2-25d62c461f3b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.112218 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.112051 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.112218 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.112109 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.112218 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.112178 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.112572 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.112553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-web-config\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.112613 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.112558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac3278cd-1342-427d-80a2-25d62c461f3b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.112869 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.112850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac3278cd-1342-427d-80a2-25d62c461f3b-config-out\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.118815 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.118796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8555k\" (UniqueName: \"kubernetes.io/projected/ac3278cd-1342-427d-80a2-25d62c461f3b-kube-api-access-8555k\") pod \"alertmanager-main-0\" (UID: \"ac3278cd-1342-427d-80a2-25d62c461f3b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.154707 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:04:13.154675 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-95875" podUID="87705952-f6c4-4c65-83a7-1a04256c34b1" Jul 20 05:04:13.175758 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.175734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 20 05:04:13.311169 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.310985 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 20 05:04:13.313452 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:04:13.313427 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3278cd_1342_427d_80a2_25d62c461f3b.slice/crio-7a1309bfcc35ed2d1f889cd3ab22f47f15f57858f1d502e3d4bc7d81f4941233 WatchSource:0}: Error finding container 7a1309bfcc35ed2d1f889cd3ab22f47f15f57858f1d502e3d4bc7d81f4941233: Status 404 returned error can't find the container with id 7a1309bfcc35ed2d1f889cd3ab22f47f15f57858f1d502e3d4bc7d81f4941233 Jul 20 05:04:13.531386 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.531344 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gh5fb" event={"ID":"b04d0856-7c69-4200-bf79-ac7bab49f604","Type":"ContainerStarted","Data":"9214b003ea312360a2cbf23f796f1f35b1c1a62133b099d39a3cd5f82df5aa0c"} Jul 20 05:04:13.532894 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.532859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac3278cd-1342-427d-80a2-25d62c461f3b","Type":"ContainerStarted","Data":"7a1309bfcc35ed2d1f889cd3ab22f47f15f57858f1d502e3d4bc7d81f4941233"} Jul 20 05:04:13.532894 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:13.532890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95875" Jul 20 05:04:14.538669 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:14.538620 2579 generic.go:356] "Generic (PLEG): container finished" podID="b04d0856-7c69-4200-bf79-ac7bab49f604" containerID="23c4d79813fc4c3d324198fc5cb73125eae339a4bbd7ef7539d33ad4392c6e42" exitCode=0 Jul 20 05:04:14.539196 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:14.538715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gh5fb" event={"ID":"b04d0856-7c69-4200-bf79-ac7bab49f604","Type":"ContainerDied","Data":"23c4d79813fc4c3d324198fc5cb73125eae339a4bbd7ef7539d33ad4392c6e42"} Jul 20 05:04:15.544150 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:15.544111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gh5fb" event={"ID":"b04d0856-7c69-4200-bf79-ac7bab49f604","Type":"ContainerStarted","Data":"73ff1abe394d5b79e21429ce708af57a2b4f73081454d8d537486f8a5da69987"} Jul 20 05:04:15.544575 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:15.544173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gh5fb" event={"ID":"b04d0856-7c69-4200-bf79-ac7bab49f604","Type":"ContainerStarted","Data":"73f4b10200d933b5647822212f78cfdd8df9f70599cb546ee56ca828c8048a6d"} Jul 20 05:04:15.545487 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:15.545463 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac3278cd-1342-427d-80a2-25d62c461f3b" containerID="bc173275ba9b1fe40debfaccc813a2803015d8056c0ebab9297c44b5236b0115" exitCode=0 Jul 20 05:04:15.545581 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:15.545530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac3278cd-1342-427d-80a2-25d62c461f3b","Type":"ContainerDied","Data":"bc173275ba9b1fe40debfaccc813a2803015d8056c0ebab9297c44b5236b0115"} Jul 20 05:04:15.561481 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:15.561431 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gh5fb" podStartSLOduration=3.434564005 podStartE2EDuration="4.561415717s" totalImagesPullingTime="1.063256578s" totalInitContainerRuntime="63.595134ms" isStatefulPod=true podCreationTimestamp="2026-07-20 05:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:04:15.560285242 +0000 UTC m=+284.829768928" watchObservedRunningTime="2026-07-20 05:04:15.561415717 +0000 UTC m=+284.830899433" Jul 20 05:04:16.437326 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.437285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:04:16.439938 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.439907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/87705952-f6c4-4c65-83a7-1a04256c34b1-metrics-tls\") pod \"dns-default-95875\" (UID: \"87705952-f6c4-4c65-83a7-1a04256c34b1\") " pod="openshift-dns/dns-default-95875" Jul 20 05:04:16.536582 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.536550 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jlw9p\"" Jul 20 05:04:16.544633 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.544603 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-95875" Jul 20 05:04:16.616148 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.616096 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s"] Jul 20 05:04:16.645646 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.643159 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s"] Jul 20 05:04:16.645646 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.643284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:16.646252 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.646224 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fb4rt\"" Jul 20 05:04:16.646545 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.646492 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 20 05:04:16.728861 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.728757 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-95875"] Jul 20 05:04:16.731787 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:04:16.731754 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87705952_f6c4_4c65_83a7_1a04256c34b1.slice/crio-8178dfec3d2227165a848d7ccb78384bd5c983a6cb0ee0e99a6f9daa80ec0532 WatchSource:0}: Error finding container 8178dfec3d2227165a848d7ccb78384bd5c983a6cb0ee0e99a6f9daa80ec0532: Status 404 returned error can't find the container with id 8178dfec3d2227165a848d7ccb78384bd5c983a6cb0ee0e99a6f9daa80ec0532 Jul 20 05:04:16.747023 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.746999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b92cdf5c-6254-4b35-bb4a-170cb1971ad7-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p4t9s\" (UID: \"b92cdf5c-6254-4b35-bb4a-170cb1971ad7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:16.747147 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.747076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr4sm\" (UniqueName: \"kubernetes.io/projected/b92cdf5c-6254-4b35-bb4a-170cb1971ad7-kube-api-access-pr4sm\") pod \"monitoring-plugin-7b7b7bc8bf-p4t9s\" (UID: \"b92cdf5c-6254-4b35-bb4a-170cb1971ad7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:16.848070 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.848022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b92cdf5c-6254-4b35-bb4a-170cb1971ad7-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p4t9s\" (UID: \"b92cdf5c-6254-4b35-bb4a-170cb1971ad7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:16.848222 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.848107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr4sm\" (UniqueName: \"kubernetes.io/projected/b92cdf5c-6254-4b35-bb4a-170cb1971ad7-kube-api-access-pr4sm\") pod \"monitoring-plugin-7b7b7bc8bf-p4t9s\" (UID: \"b92cdf5c-6254-4b35-bb4a-170cb1971ad7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:16.850520 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.850491 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b92cdf5c-6254-4b35-bb4a-170cb1971ad7-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p4t9s\" (UID: \"b92cdf5c-6254-4b35-bb4a-170cb1971ad7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:16.856052 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.856027 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr4sm\" (UniqueName: \"kubernetes.io/projected/b92cdf5c-6254-4b35-bb4a-170cb1971ad7-kube-api-access-pr4sm\") pod \"monitoring-plugin-7b7b7bc8bf-p4t9s\" (UID: \"b92cdf5c-6254-4b35-bb4a-170cb1971ad7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:16.956301 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:16.956261 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:17.311415 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:17.311380 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s"] Jul 20 05:04:17.314244 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:04:17.314217 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92cdf5c_6254_4b35_bb4a_170cb1971ad7.slice/crio-59e288753ab0cbe8090d282f5e9bbc436ad87ca7d2410395d46ee66cc5ba112b WatchSource:0}: Error finding container 59e288753ab0cbe8090d282f5e9bbc436ad87ca7d2410395d46ee66cc5ba112b: Status 404 returned error can't find the container with id 59e288753ab0cbe8090d282f5e9bbc436ad87ca7d2410395d46ee66cc5ba112b Jul 20 05:04:17.554319 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:17.554286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" event={"ID":"b92cdf5c-6254-4b35-bb4a-170cb1971ad7","Type":"ContainerStarted","Data":"59e288753ab0cbe8090d282f5e9bbc436ad87ca7d2410395d46ee66cc5ba112b"} Jul 20 05:04:17.555296 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:17.555274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95875" event={"ID":"87705952-f6c4-4c65-83a7-1a04256c34b1","Type":"ContainerStarted","Data":"8178dfec3d2227165a848d7ccb78384bd5c983a6cb0ee0e99a6f9daa80ec0532"} Jul 20 05:04:18.563302 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:18.563266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac3278cd-1342-427d-80a2-25d62c461f3b","Type":"ContainerStarted","Data":"33dc44d8220063db4a3abd262e8759c7b5f1f8a567ccf961df7b6f560fbb4526"} Jul 20 05:04:18.563700 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:18.563368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac3278cd-1342-427d-80a2-25d62c461f3b","Type":"ContainerStarted","Data":"46c7ce914c1ae0eb52eec15cdc30f8ac9a4811b1dc8f131afaf20f7bd40cd294"} Jul 20 05:04:18.563700 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:18.563407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac3278cd-1342-427d-80a2-25d62c461f3b","Type":"ContainerStarted","Data":"3e38f32f738f280eced32377744cf625b3f625b13d9ea23a0dde8c74a021a30e"} Jul 20 05:04:18.563700 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:18.563435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac3278cd-1342-427d-80a2-25d62c461f3b","Type":"ContainerStarted","Data":"5920388c909295643370d3f970e06e38a70bb93c805f2aed118f66c083551b6a"} Jul 20 05:04:18.563700 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:18.563476 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac3278cd-1342-427d-80a2-25d62c461f3b","Type":"ContainerStarted","Data":"b85fb8f4ddb0f9fb3cd0dc00642a80d913cf50f83380af4924484aa211b5356c"} Jul 20 05:04:20.571127 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.571087 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" event={"ID":"b92cdf5c-6254-4b35-bb4a-170cb1971ad7","Type":"ContainerStarted","Data":"fb896f5edfbdc2b2d34deeeca653ee1de1549f8f39bef5bc0a5c4dd7166a65cc"} Jul 20 05:04:20.571568 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.571157 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:20.574802 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.574770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac3278cd-1342-427d-80a2-25d62c461f3b","Type":"ContainerStarted","Data":"1c96218bac883308dfa2a20b7a83a21869ff1d9cd0574e4acd34e43153bb35f3"} Jul 20 05:04:20.576527 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.576502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95875" event={"ID":"87705952-f6c4-4c65-83a7-1a04256c34b1","Type":"ContainerStarted","Data":"a06b63cc7adeaaf96565dea276b084cd321a283b595e9523511ac2e805d40273"} Jul 20 05:04:20.576652 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.576551 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-95875" event={"ID":"87705952-f6c4-4c65-83a7-1a04256c34b1","Type":"ContainerStarted","Data":"aa0cca9f6f1ece79c76681f3eee989e018dce5af6cc6a15b224699cb37f75eb5"} Jul 20 05:04:20.576652 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.576589 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-95875" Jul 20 05:04:20.576797 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.576783 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" Jul 20 05:04:20.586409 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.586350 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p4t9s" podStartSLOduration=2.284781503 podStartE2EDuration="4.586336776s" totalImagesPullingTime="2.301555273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:04:20.584978406 +0000 UTC m=+289.854462091" watchObservedRunningTime="2026-07-20 05:04:20.586336776 +0000 UTC m=+289.855820460" Jul 20 05:04:20.609852 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.609797 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.787471562 podStartE2EDuration="8.609780186s" totalImagesPullingTime="5.809313004s" totalInitContainerRuntime="12.99562ms" isStatefulPod=false podCreationTimestamp="2026-07-20 05:04:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:04:20.607398538 +0000 UTC m=+289.876882264" watchObservedRunningTime="2026-07-20 05:04:20.609780186 +0000 UTC m=+289.879263849" Jul 20 05:04:20.624667 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:20.624614 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-95875" podStartSLOduration=251.74429729 podStartE2EDuration="4m14.624597116s" totalImagesPullingTime="2.880299826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:04:20.623703274 +0000 UTC m=+289.893186961" watchObservedRunningTime="2026-07-20 05:04:20.624597116 +0000 UTC m=+289.894080856" Jul 20 05:04:22.883649 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:22.883610 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" podUID="53e35fb0-4c2d-47e9-a405-57db73feadce" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 05:04:30.582127 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:30.582090 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-95875" Jul 20 05:04:31.067172 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:31.067148 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 20 05:04:32.883743 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:32.883705 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" podUID="53e35fb0-4c2d-47e9-a405-57db73feadce" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 20 05:04:32.886138 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:32.883780 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" Jul 20 05:04:32.886138 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:32.884240 2579 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f3f23fd4733389bf32ec405497c90c258fcd6363925ed15391ff5192f0114bea"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 20 05:04:32.886138 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:32.884293 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" podUID="53e35fb0-4c2d-47e9-a405-57db73feadce" containerName="service-proxy" containerID="cri-o://f3f23fd4733389bf32ec405497c90c258fcd6363925ed15391ff5192f0114bea" gracePeriod=30 Jul 20 05:04:33.624376 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:33.624342 2579 generic.go:356] "Generic (PLEG): container finished" podID="53e35fb0-4c2d-47e9-a405-57db73feadce" containerID="f3f23fd4733389bf32ec405497c90c258fcd6363925ed15391ff5192f0114bea" exitCode=2 Jul 20 05:04:33.624376 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:33.624381 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" event={"ID":"53e35fb0-4c2d-47e9-a405-57db73feadce","Type":"ContainerDied","Data":"f3f23fd4733389bf32ec405497c90c258fcd6363925ed15391ff5192f0114bea"} Jul 20 05:04:33.624582 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:33.624415 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-775dcbdc4-vtdrn" event={"ID":"53e35fb0-4c2d-47e9-a405-57db73feadce","Type":"ContainerStarted","Data":"479c03eace93f9de325a7496872601c355a55e751b71e8a774139d97ec91b008"} Jul 20 05:04:37.653601 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.653552 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dc58cd79c-skjdb" podUID="3373bdb1-ea18-42e3-a929-485044a77fe6" containerName="console" containerID="cri-o://9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495" gracePeriod=15 Jul 20 05:04:37.885572 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.885543 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:37.900703 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.900681 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-service-ca\") pod \"3373bdb1-ea18-42e3-a929-485044a77fe6\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " Jul 20 05:04:37.900835 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.900715 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5d5c\" (UniqueName: \"kubernetes.io/projected/3373bdb1-ea18-42e3-a929-485044a77fe6-kube-api-access-j5d5c\") pod \"3373bdb1-ea18-42e3-a929-485044a77fe6\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " Jul 20 05:04:37.900835 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.900734 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-serving-cert\") pod \"3373bdb1-ea18-42e3-a929-485044a77fe6\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " Jul 20 05:04:37.900835 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.900761 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-oauth-config\") pod \"3373bdb1-ea18-42e3-a929-485044a77fe6\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " Jul 20 05:04:37.900835 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.900795 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-console-config\") pod \"3373bdb1-ea18-42e3-a929-485044a77fe6\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " Jul 20 05:04:37.900835 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.900815 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-oauth-serving-cert\") pod \"3373bdb1-ea18-42e3-a929-485044a77fe6\" (UID: \"3373bdb1-ea18-42e3-a929-485044a77fe6\") " Jul 20 05:04:37.901216 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.901191 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-service-ca" (OuterVolumeSpecName: "service-ca") pod "3373bdb1-ea18-42e3-a929-485044a77fe6" (UID: "3373bdb1-ea18-42e3-a929-485044a77fe6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:04:37.901290 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.901211 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-console-config" (OuterVolumeSpecName: "console-config") pod "3373bdb1-ea18-42e3-a929-485044a77fe6" (UID: "3373bdb1-ea18-42e3-a929-485044a77fe6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:04:37.901290 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.901251 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3373bdb1-ea18-42e3-a929-485044a77fe6" (UID: "3373bdb1-ea18-42e3-a929-485044a77fe6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:04:37.903359 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.903313 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3373bdb1-ea18-42e3-a929-485044a77fe6" (UID: "3373bdb1-ea18-42e3-a929-485044a77fe6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 05:04:37.903359 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.903314 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3373bdb1-ea18-42e3-a929-485044a77fe6-kube-api-access-j5d5c" (OuterVolumeSpecName: "kube-api-access-j5d5c") pod "3373bdb1-ea18-42e3-a929-485044a77fe6" (UID: "3373bdb1-ea18-42e3-a929-485044a77fe6"). InnerVolumeSpecName "kube-api-access-j5d5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:04:37.903542 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:37.903362 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3373bdb1-ea18-42e3-a929-485044a77fe6" (UID: "3373bdb1-ea18-42e3-a929-485044a77fe6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 05:04:38.001624 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.001570 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-service-ca\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:04:38.001624 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.001619 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5d5c\" (UniqueName: \"kubernetes.io/projected/3373bdb1-ea18-42e3-a929-485044a77fe6-kube-api-access-j5d5c\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:04:38.001624 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.001634 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-serving-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:04:38.001863 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.001646 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3373bdb1-ea18-42e3-a929-485044a77fe6-console-oauth-config\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:04:38.001863 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.001659 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-console-config\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:04:38.001863 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.001672 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3373bdb1-ea18-42e3-a929-485044a77fe6-oauth-serving-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:04:38.643718 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.643686 2579 generic.go:356] "Generic (PLEG): container finished" podID="3373bdb1-ea18-42e3-a929-485044a77fe6" containerID="9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495" exitCode=0 Jul 20 05:04:38.643925 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.643738 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc58cd79c-skjdb" Jul 20 05:04:38.643925 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.643766 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc58cd79c-skjdb" event={"ID":"3373bdb1-ea18-42e3-a929-485044a77fe6","Type":"ContainerDied","Data":"9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495"} Jul 20 05:04:38.643925 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.643816 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc58cd79c-skjdb" event={"ID":"3373bdb1-ea18-42e3-a929-485044a77fe6","Type":"ContainerDied","Data":"4505b98be56eaf447a89913ab23a6b0c0083d977fed87520a28a8d714dc381ea"} Jul 20 05:04:38.643925 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.643838 2579 scope.go:117] "RemoveContainer" containerID="9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495" Jul 20 05:04:38.653100 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.653083 2579 scope.go:117] "RemoveContainer" containerID="9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495" Jul 20 05:04:38.653392 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:04:38.653375 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495\": container with ID starting with 9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495 not found: ID does not exist" containerID="9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495" Jul 20 05:04:38.653456 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.653397 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495"} err="failed to get container status \"9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495\": rpc error: code = NotFound desc = could not find container \"9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495\": container with ID starting with 9e89fe997f3173dedfca0a867d71660762c35c68e7b975c79b66be55fc32d495 not found: ID does not exist" Jul 20 05:04:38.664090 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.664050 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc58cd79c-skjdb"] Jul 20 05:04:38.669453 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:38.669431 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dc58cd79c-skjdb"] Jul 20 05:04:39.154133 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:04:39.154099 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3373bdb1-ea18-42e3-a929-485044a77fe6" path="/var/lib/kubelet/pods/3373bdb1-ea18-42e3-a929-485044a77fe6/volumes" Jul 20 05:05:29.834090 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.833990 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9664bb7f6-pcvpq"] Jul 20 05:05:29.834592 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.834267 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3373bdb1-ea18-42e3-a929-485044a77fe6" containerName="console" Jul 20 05:05:29.834592 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.834279 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3373bdb1-ea18-42e3-a929-485044a77fe6" containerName="console" Jul 20 05:05:29.836758 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.834827 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3373bdb1-ea18-42e3-a929-485044a77fe6" containerName="console" Jul 20 05:05:29.838902 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.838874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.841556 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.841533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 20 05:05:29.841556 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.841555 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 20 05:05:29.841718 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.841628 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 20 05:05:29.841718 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.841676 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 20 05:05:29.841718 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.841696 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 20 05:05:29.842029 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.842004 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 20 05:05:29.842120 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.842038 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-25rk9\"" Jul 20 05:05:29.842913 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.842897 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 20 05:05:29.847563 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.847539 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 20 05:05:29.848775 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.848754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9664bb7f6-pcvpq"] Jul 20 05:05:29.860946 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.860920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-service-ca\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.861111 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.860952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnwp\" (UniqueName: \"kubernetes.io/projected/6dbf11ce-1264-429c-af83-ba48f946ad7e-kube-api-access-jhnwp\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.861111 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.860979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-oauth-config\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.861111 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.860995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-config\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.861111 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.861022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-serving-cert\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.861111 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.861044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-trusted-ca-bundle\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.861271 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.861133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-oauth-serving-cert\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.962296 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.962263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-oauth-serving-cert\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.962479 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.962309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-service-ca\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.962479 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.962335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnwp\" (UniqueName: \"kubernetes.io/projected/6dbf11ce-1264-429c-af83-ba48f946ad7e-kube-api-access-jhnwp\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.962479 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.962369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-oauth-config\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.962479 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.962390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-config\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.962479 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.962422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-serving-cert\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.962479 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.962455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-trusted-ca-bundle\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.963118 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.963054 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-oauth-serving-cert\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.963225 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.963184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-service-ca\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.963281 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.963266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-config\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.963343 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.963293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-trusted-ca-bundle\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.964923 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.964905 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-serving-cert\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.965007 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.964926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-oauth-config\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:29.970665 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:29.970643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnwp\" (UniqueName: \"kubernetes.io/projected/6dbf11ce-1264-429c-af83-ba48f946ad7e-kube-api-access-jhnwp\") pod \"console-9664bb7f6-pcvpq\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:30.149095 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:30.148954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:30.279582 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:30.279557 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9664bb7f6-pcvpq"] Jul 20 05:05:30.282110 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:05:30.282080 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbf11ce_1264_429c_af83_ba48f946ad7e.slice/crio-8728ed2091517b78c887f7b5be408446ae76587093881dced9a5087d20510e9a WatchSource:0}: Error finding container 8728ed2091517b78c887f7b5be408446ae76587093881dced9a5087d20510e9a: Status 404 returned error can't find the container with id 8728ed2091517b78c887f7b5be408446ae76587093881dced9a5087d20510e9a Jul 20 05:05:30.831594 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:30.831563 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9664bb7f6-pcvpq" event={"ID":"6dbf11ce-1264-429c-af83-ba48f946ad7e","Type":"ContainerStarted","Data":"050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80"} Jul 20 05:05:30.831778 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:30.831615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9664bb7f6-pcvpq" event={"ID":"6dbf11ce-1264-429c-af83-ba48f946ad7e","Type":"ContainerStarted","Data":"8728ed2091517b78c887f7b5be408446ae76587093881dced9a5087d20510e9a"} Jul 20 05:05:30.849915 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:30.849866 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9664bb7f6-pcvpq" podStartSLOduration=1.849851154 podStartE2EDuration="1.849851154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:05:30.848351641 +0000 UTC m=+360.117835328" watchObservedRunningTime="2026-07-20 05:05:30.849851154 +0000 UTC m=+360.119334838" Jul 20 05:05:40.150115 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:40.150078 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:40.150604 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:40.150155 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:40.154898 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:40.154874 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:05:40.869688 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:05:40.869657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:06:52.563396 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.561457 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mw9qc"] Jul 20 05:06:52.565619 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.565578 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:06:52.568676 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.568649 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 20 05:06:52.569804 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.569778 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qh8rg\"" Jul 20 05:06:52.569918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.569778 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 20 05:06:52.570120 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.570079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxv6m\" (UniqueName: \"kubernetes.io/projected/93274074-624c-4e7f-85ed-b965feadf42f-kube-api-access-pxv6m\") pod \"cert-manager-webhook-6486ffd997-mw9qc\" (UID: \"93274074-624c-4e7f-85ed-b965feadf42f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:06:52.570197 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.570182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93274074-624c-4e7f-85ed-b965feadf42f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mw9qc\" (UID: \"93274074-624c-4e7f-85ed-b965feadf42f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:06:52.574775 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.574742 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mw9qc"] Jul 20 05:06:52.671511 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.671475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxv6m\" (UniqueName: \"kubernetes.io/projected/93274074-624c-4e7f-85ed-b965feadf42f-kube-api-access-pxv6m\") pod \"cert-manager-webhook-6486ffd997-mw9qc\" (UID: \"93274074-624c-4e7f-85ed-b965feadf42f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:06:52.671722 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.671535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93274074-624c-4e7f-85ed-b965feadf42f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mw9qc\" (UID: \"93274074-624c-4e7f-85ed-b965feadf42f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:06:52.679440 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.679412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93274074-624c-4e7f-85ed-b965feadf42f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mw9qc\" (UID: \"93274074-624c-4e7f-85ed-b965feadf42f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:06:52.679711 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.679689 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxv6m\" (UniqueName: \"kubernetes.io/projected/93274074-624c-4e7f-85ed-b965feadf42f-kube-api-access-pxv6m\") pod \"cert-manager-webhook-6486ffd997-mw9qc\" (UID: \"93274074-624c-4e7f-85ed-b965feadf42f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:06:52.875558 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:52.875475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:06:53.006773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:53.006607 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mw9qc"] Jul 20 05:06:53.009522 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:06:53.009489 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93274074_624c_4e7f_85ed_b965feadf42f.slice/crio-c35c563bdacf24d627edbcb89dec7c7a87f51439508f33b0198517564e42bee9 WatchSource:0}: Error finding container c35c563bdacf24d627edbcb89dec7c7a87f51439508f33b0198517564e42bee9: Status 404 returned error can't find the container with id c35c563bdacf24d627edbcb89dec7c7a87f51439508f33b0198517564e42bee9 Jul 20 05:06:53.011451 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:53.011433 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 05:06:53.111728 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:53.111693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" event={"ID":"93274074-624c-4e7f-85ed-b965feadf42f","Type":"ContainerStarted","Data":"c35c563bdacf24d627edbcb89dec7c7a87f51439508f33b0198517564e42bee9"} Jul 20 05:06:57.127617 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:57.127576 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" event={"ID":"93274074-624c-4e7f-85ed-b965feadf42f","Type":"ContainerStarted","Data":"b72da4e1223a1eeb36b03e8ed4976f734a26c53e8408e1818c3d28db6ad1a749"} Jul 20 05:06:57.128015 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:57.127692 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:06:57.143369 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:57.143318 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" podStartSLOduration=2.006370241 podStartE2EDuration="5.143303309s" totalImagesPullingTime="3.136933068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:06:57.141858397 +0000 UTC m=+446.411342083" watchObservedRunningTime="2026-07-20 05:06:57.143303309 +0000 UTC m=+446.412786994" Jul 20 05:06:59.400079 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.400028 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2"] Jul 20 05:06:59.403230 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.403213 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" Jul 20 05:06:59.405790 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.405760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kbdj4\"" Jul 20 05:06:59.405790 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.405779 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 20 05:06:59.406904 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.406885 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 20 05:06:59.412234 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.412208 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2"] Jul 20 05:06:59.423486 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.423450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f6ad10f-f268-4c52-b51a-4b7032f382c1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rhnd2\" (UID: \"0f6ad10f-f268-4c52-b51a-4b7032f382c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" Jul 20 05:06:59.423486 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.423482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc449\" (UniqueName: \"kubernetes.io/projected/0f6ad10f-f268-4c52-b51a-4b7032f382c1-kube-api-access-mc449\") pod \"openshift-lws-operator-fd8ccff4c-rhnd2\" (UID: \"0f6ad10f-f268-4c52-b51a-4b7032f382c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" Jul 20 05:06:59.523840 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.523810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f6ad10f-f268-4c52-b51a-4b7032f382c1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rhnd2\" (UID: \"0f6ad10f-f268-4c52-b51a-4b7032f382c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" Jul 20 05:06:59.523840 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.523846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc449\" (UniqueName: \"kubernetes.io/projected/0f6ad10f-f268-4c52-b51a-4b7032f382c1-kube-api-access-mc449\") pod \"openshift-lws-operator-fd8ccff4c-rhnd2\" (UID: \"0f6ad10f-f268-4c52-b51a-4b7032f382c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" Jul 20 05:06:59.524300 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.524279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f6ad10f-f268-4c52-b51a-4b7032f382c1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rhnd2\" (UID: \"0f6ad10f-f268-4c52-b51a-4b7032f382c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" Jul 20 05:06:59.531675 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.531647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc449\" (UniqueName: \"kubernetes.io/projected/0f6ad10f-f268-4c52-b51a-4b7032f382c1-kube-api-access-mc449\") pod \"openshift-lws-operator-fd8ccff4c-rhnd2\" (UID: \"0f6ad10f-f268-4c52-b51a-4b7032f382c1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" Jul 20 05:06:59.712698 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.712612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" Jul 20 05:06:59.838994 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:06:59.838946 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2"] Jul 20 05:06:59.841688 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:06:59.841657 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f6ad10f_f268_4c52_b51a_4b7032f382c1.slice/crio-57c45f9f2eed3efd3f5cc653cd99d3745e7377f110e05844ea451de5c9c5443b WatchSource:0}: Error finding container 57c45f9f2eed3efd3f5cc653cd99d3745e7377f110e05844ea451de5c9c5443b: Status 404 returned error can't find the container with id 57c45f9f2eed3efd3f5cc653cd99d3745e7377f110e05844ea451de5c9c5443b Jul 20 05:07:00.138844 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:00.138809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" event={"ID":"0f6ad10f-f268-4c52-b51a-4b7032f382c1","Type":"ContainerStarted","Data":"57c45f9f2eed3efd3f5cc653cd99d3745e7377f110e05844ea451de5c9c5443b"} Jul 20 05:07:03.133601 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:03.133567 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mw9qc" Jul 20 05:07:24.225150 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:24.225113 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" event={"ID":"0f6ad10f-f268-4c52-b51a-4b7032f382c1","Type":"ContainerStarted","Data":"200ae28431ff3c96f76afdf0d1e7faa6a1d3fe10e38028783b116742e950a644"} Jul 20 05:07:24.240692 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:24.240638 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rhnd2" podStartSLOduration=0.985164057 podStartE2EDuration="25.240623782s" totalImagesPullingTime="24.255459725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:07:24.238975645 +0000 UTC m=+473.508459334" watchObservedRunningTime="2026-07-20 05:07:24.240623782 +0000 UTC m=+473.510107466" Jul 20 05:07:39.760137 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.760102 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv"] Jul 20 05:07:39.763422 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.763404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.766120 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.766096 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 20 05:07:39.766239 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.766212 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 20 05:07:39.766307 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.766234 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5gbtj\"" Jul 20 05:07:39.766307 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.766255 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 20 05:07:39.766307 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.766293 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 20 05:07:39.778129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.778106 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv"] Jul 20 05:07:39.837775 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.837731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79w9\" (UniqueName: \"kubernetes.io/projected/91136ba1-8512-40ca-8958-1f7b33aed995-kube-api-access-z79w9\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.837966 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.837782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91136ba1-8512-40ca-8958-1f7b33aed995-tmp\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.837966 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.837808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91136ba1-8512-40ca-8958-1f7b33aed995-webhook-cert\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.837966 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.837876 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/91136ba1-8512-40ca-8958-1f7b33aed995-manifests\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.837966 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.837921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/91136ba1-8512-40ca-8958-1f7b33aed995-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.938834 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.938793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91136ba1-8512-40ca-8958-1f7b33aed995-tmp\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.938834 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.938837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91136ba1-8512-40ca-8958-1f7b33aed995-webhook-cert\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.939121 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.938889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/91136ba1-8512-40ca-8958-1f7b33aed995-manifests\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.939121 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.938933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/91136ba1-8512-40ca-8958-1f7b33aed995-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.939121 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.938983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79w9\" (UniqueName: \"kubernetes.io/projected/91136ba1-8512-40ca-8958-1f7b33aed995-kube-api-access-z79w9\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.939308 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.939284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/91136ba1-8512-40ca-8958-1f7b33aed995-manifests\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.941181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.941159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91136ba1-8512-40ca-8958-1f7b33aed995-tmp\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.941466 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.941446 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/91136ba1-8512-40ca-8958-1f7b33aed995-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.941531 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.941462 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91136ba1-8512-40ca-8958-1f7b33aed995-webhook-cert\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:39.950263 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:39.950242 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79w9\" (UniqueName: \"kubernetes.io/projected/91136ba1-8512-40ca-8958-1f7b33aed995-kube-api-access-z79w9\") pod \"opendatahub-operator-controller-manager-54dff8fcf7-kzrfv\" (UID: \"91136ba1-8512-40ca-8958-1f7b33aed995\") " pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:40.073892 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:40.073863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:40.206540 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:40.206505 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv"] Jul 20 05:07:40.210012 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:07:40.209987 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91136ba1_8512_40ca_8958_1f7b33aed995.slice/crio-d1dc9f8101bbc00ddbbe5f68945045426c04aed8608a0dab5bc5f7502e5cf2a1 WatchSource:0}: Error finding container d1dc9f8101bbc00ddbbe5f68945045426c04aed8608a0dab5bc5f7502e5cf2a1: Status 404 returned error can't find the container with id d1dc9f8101bbc00ddbbe5f68945045426c04aed8608a0dab5bc5f7502e5cf2a1 Jul 20 05:07:40.284252 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:40.284213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" event={"ID":"91136ba1-8512-40ca-8958-1f7b33aed995","Type":"ContainerStarted","Data":"d1dc9f8101bbc00ddbbe5f68945045426c04aed8608a0dab5bc5f7502e5cf2a1"} Jul 20 05:07:43.297856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:43.297821 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" event={"ID":"91136ba1-8512-40ca-8958-1f7b33aed995","Type":"ContainerStarted","Data":"c3ff8befd286febcf1f9dcd6490e8811f6b6101d60c455c2f58c3402a03a444b"} Jul 20 05:07:44.303364 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:44.303331 2579 generic.go:356] "Generic (PLEG): container finished" podID="91136ba1-8512-40ca-8958-1f7b33aed995" containerID="c3ff8befd286febcf1f9dcd6490e8811f6b6101d60c455c2f58c3402a03a444b" exitCode=0 Jul 20 05:07:44.303751 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:44.303370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" event={"ID":"91136ba1-8512-40ca-8958-1f7b33aed995","Type":"ContainerDied","Data":"c3ff8befd286febcf1f9dcd6490e8811f6b6101d60c455c2f58c3402a03a444b"} Jul 20 05:07:45.309908 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:45.309873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" event={"ID":"91136ba1-8512-40ca-8958-1f7b33aed995","Type":"ContainerStarted","Data":"afa29d583997c46b3e90c2481c894d4d1adb5fb3dcec74909063f6a7fc5223a3"} Jul 20 05:07:45.310422 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:45.310016 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:45.331247 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:45.331197 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" podStartSLOduration=2.051547795 podStartE2EDuration="6.331180921s" totalImagesPullingTime="3.246729313s" totalInitContainerRuntime="1.032903813s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:07:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:07:45.329604867 +0000 UTC m=+494.599088551" watchObservedRunningTime="2026-07-20 05:07:45.331180921 +0000 UTC m=+494.600664652" Jul 20 05:07:56.317575 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:56.317545 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54dff8fcf7-kzrfv" Jul 20 05:07:58.903716 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:58.903677 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pqmcj"] Jul 20 05:07:58.950293 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:58.950260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pqmcj"] Jul 20 05:07:58.950456 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:58.950370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:07:58.952954 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:58.952927 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-q6gkp\"" Jul 20 05:07:58.953068 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:58.952936 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 20 05:07:59.080680 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.080645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhp7s\" (UniqueName: \"kubernetes.io/projected/a6e0da71-dac3-475e-9fa6-5df2be58f7f1-kube-api-access-vhp7s\") pod \"odh-model-controller-7998c67dc4-pqmcj\" (UID: \"a6e0da71-dac3-475e-9fa6-5df2be58f7f1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:07:59.080878 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.080717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e0da71-dac3-475e-9fa6-5df2be58f7f1-cert\") pod \"odh-model-controller-7998c67dc4-pqmcj\" (UID: \"a6e0da71-dac3-475e-9fa6-5df2be58f7f1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:07:59.182012 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.181916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhp7s\" (UniqueName: \"kubernetes.io/projected/a6e0da71-dac3-475e-9fa6-5df2be58f7f1-kube-api-access-vhp7s\") pod \"odh-model-controller-7998c67dc4-pqmcj\" (UID: \"a6e0da71-dac3-475e-9fa6-5df2be58f7f1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:07:59.182179 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.182022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e0da71-dac3-475e-9fa6-5df2be58f7f1-cert\") pod \"odh-model-controller-7998c67dc4-pqmcj\" (UID: \"a6e0da71-dac3-475e-9fa6-5df2be58f7f1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:07:59.182222 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:07:59.182187 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 20 05:07:59.182301 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:07:59.182289 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e0da71-dac3-475e-9fa6-5df2be58f7f1-cert podName:a6e0da71-dac3-475e-9fa6-5df2be58f7f1 nodeName:}" failed. No retries permitted until 2026-07-20 05:07:59.682267508 +0000 UTC m=+508.951751174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a6e0da71-dac3-475e-9fa6-5df2be58f7f1-cert") pod "odh-model-controller-7998c67dc4-pqmcj" (UID: "a6e0da71-dac3-475e-9fa6-5df2be58f7f1") : secret "odh-model-controller-webhook-cert" not found Jul 20 05:07:59.190568 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.190532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhp7s\" (UniqueName: \"kubernetes.io/projected/a6e0da71-dac3-475e-9fa6-5df2be58f7f1-kube-api-access-vhp7s\") pod \"odh-model-controller-7998c67dc4-pqmcj\" (UID: \"a6e0da71-dac3-475e-9fa6-5df2be58f7f1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:07:59.519511 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.519475 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vxr6x"] Jul 20 05:07:59.561861 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.561831 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vxr6x"] Jul 20 05:07:59.562020 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.561949 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:07:59.564300 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.564277 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 20 05:07:59.686595 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.686564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8619931b-64ab-4871-9ff8-314504bb263a-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxr6x\" (UID: \"8619931b-64ab-4871-9ff8-314504bb263a\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:07:59.686747 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.686684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms8tm\" (UniqueName: \"kubernetes.io/projected/8619931b-64ab-4871-9ff8-314504bb263a-kube-api-access-ms8tm\") pod \"model-serving-api-54d89bd5fc-vxr6x\" (UID: \"8619931b-64ab-4871-9ff8-314504bb263a\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:07:59.686747 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.686729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e0da71-dac3-475e-9fa6-5df2be58f7f1-cert\") pod \"odh-model-controller-7998c67dc4-pqmcj\" (UID: \"a6e0da71-dac3-475e-9fa6-5df2be58f7f1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:07:59.689229 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.689200 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e0da71-dac3-475e-9fa6-5df2be58f7f1-cert\") pod \"odh-model-controller-7998c67dc4-pqmcj\" (UID: \"a6e0da71-dac3-475e-9fa6-5df2be58f7f1\") " pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:07:59.787816 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.787731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms8tm\" (UniqueName: \"kubernetes.io/projected/8619931b-64ab-4871-9ff8-314504bb263a-kube-api-access-ms8tm\") pod \"model-serving-api-54d89bd5fc-vxr6x\" (UID: \"8619931b-64ab-4871-9ff8-314504bb263a\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:07:59.787816 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.787776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8619931b-64ab-4871-9ff8-314504bb263a-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxr6x\" (UID: \"8619931b-64ab-4871-9ff8-314504bb263a\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:07:59.790280 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.790253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8619931b-64ab-4871-9ff8-314504bb263a-tls-certs\") pod \"model-serving-api-54d89bd5fc-vxr6x\" (UID: \"8619931b-64ab-4871-9ff8-314504bb263a\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:07:59.796271 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.796246 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms8tm\" (UniqueName: \"kubernetes.io/projected/8619931b-64ab-4871-9ff8-314504bb263a-kube-api-access-ms8tm\") pod \"model-serving-api-54d89bd5fc-vxr6x\" (UID: \"8619931b-64ab-4871-9ff8-314504bb263a\") " pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:07:59.860556 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.860515 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:07:59.872752 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:07:59.872714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:08:00.047689 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:00.047664 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-pqmcj"] Jul 20 05:08:00.050331 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:08:00.050303 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e0da71_dac3_475e_9fa6_5df2be58f7f1.slice/crio-8abb9817d99310216fd9118fb781f331028d1684a6177f780a741395302194c2 WatchSource:0}: Error finding container 8abb9817d99310216fd9118fb781f331028d1684a6177f780a741395302194c2: Status 404 returned error can't find the container with id 8abb9817d99310216fd9118fb781f331028d1684a6177f780a741395302194c2 Jul 20 05:08:00.059749 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:00.059706 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vxr6x"] Jul 20 05:08:00.062094 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:08:00.062070 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8619931b_64ab_4871_9ff8_314504bb263a.slice/crio-b48ae738be0ace885de8108f80de173054fc722051546ada0da3e7c48a7171d7 WatchSource:0}: Error finding container b48ae738be0ace885de8108f80de173054fc722051546ada0da3e7c48a7171d7: Status 404 returned error can't find the container with id b48ae738be0ace885de8108f80de173054fc722051546ada0da3e7c48a7171d7 Jul 20 05:08:00.372571 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:00.372456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" event={"ID":"8619931b-64ab-4871-9ff8-314504bb263a","Type":"ContainerStarted","Data":"b48ae738be0ace885de8108f80de173054fc722051546ada0da3e7c48a7171d7"} Jul 20 05:08:00.373742 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:00.373711 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" event={"ID":"a6e0da71-dac3-475e-9fa6-5df2be58f7f1","Type":"ContainerStarted","Data":"8abb9817d99310216fd9118fb781f331028d1684a6177f780a741395302194c2"} Jul 20 05:08:03.397310 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:03.397203 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" event={"ID":"8619931b-64ab-4871-9ff8-314504bb263a","Type":"ContainerStarted","Data":"72d8db968cb65afbb765bc13f0c0cee03fa4dd2e00a53b00631d2ca7da45edca"} Jul 20 05:08:03.397310 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:03.397281 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:08:03.398867 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:03.398835 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" event={"ID":"a6e0da71-dac3-475e-9fa6-5df2be58f7f1","Type":"ContainerStarted","Data":"06ecf1ef0eccf2d5d551e38ac149a96d9e9e1f187f710535f353383b947d0645"} Jul 20 05:08:03.398998 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:03.398901 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:08:03.414506 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:03.414455 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" podStartSLOduration=1.373890976 podStartE2EDuration="4.414435959s" totalImagesPullingTime="3.040544983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:08:03.413323172 +0000 UTC m=+512.682806859" watchObservedRunningTime="2026-07-20 05:08:03.414435959 +0000 UTC m=+512.683919646" Jul 20 05:08:03.428771 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:03.428709 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" podStartSLOduration=2.329089467 podStartE2EDuration="5.428690433s" totalImagesPullingTime="3.099600966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:08:03.427902193 +0000 UTC m=+512.697385879" watchObservedRunningTime="2026-07-20 05:08:03.428690433 +0000 UTC m=+512.698174123" Jul 20 05:08:08.359965 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.359925 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4"] Jul 20 05:08:08.363314 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.363293 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:08.366610 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.366590 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-gnlkq\"" Jul 20 05:08:08.366701 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.366664 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 20 05:08:08.379582 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.379556 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4"] Jul 20 05:08:08.386149 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.386109 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq"] Jul 20 05:08:08.389944 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.389921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:08.393271 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.393251 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 20 05:08:08.394180 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.394160 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-c5jb4\"" Jul 20 05:08:08.430005 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.429970 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq"] Jul 20 05:08:08.459101 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.459045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33b52d2a-b130-4f0f-b80c-7f698cb71ec7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gnfcq\" (UID: \"33b52d2a-b130-4f0f-b80c-7f698cb71ec7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:08.459257 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.459133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e3649c-b5e4-451d-936c-b47e8baaeb2e-cert\") pod \"kserve-controller-manager-d6f4b65b8-bgkb4\" (UID: \"a6e3649c-b5e4-451d-936c-b47e8baaeb2e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:08.459257 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.459165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjc2\" (UniqueName: \"kubernetes.io/projected/a6e3649c-b5e4-451d-936c-b47e8baaeb2e-kube-api-access-7pjc2\") pod \"kserve-controller-manager-d6f4b65b8-bgkb4\" (UID: \"a6e3649c-b5e4-451d-936c-b47e8baaeb2e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:08.459257 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.459199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvsc\" (UniqueName: \"kubernetes.io/projected/33b52d2a-b130-4f0f-b80c-7f698cb71ec7-kube-api-access-8jvsc\") pod \"llmisvc-controller-manager-777c6bd7d5-gnfcq\" (UID: \"33b52d2a-b130-4f0f-b80c-7f698cb71ec7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:08.560378 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.560340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvsc\" (UniqueName: \"kubernetes.io/projected/33b52d2a-b130-4f0f-b80c-7f698cb71ec7-kube-api-access-8jvsc\") pod \"llmisvc-controller-manager-777c6bd7d5-gnfcq\" (UID: \"33b52d2a-b130-4f0f-b80c-7f698cb71ec7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:08.560520 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.560421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33b52d2a-b130-4f0f-b80c-7f698cb71ec7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gnfcq\" (UID: \"33b52d2a-b130-4f0f-b80c-7f698cb71ec7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:08.560520 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.560458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e3649c-b5e4-451d-936c-b47e8baaeb2e-cert\") pod \"kserve-controller-manager-d6f4b65b8-bgkb4\" (UID: \"a6e3649c-b5e4-451d-936c-b47e8baaeb2e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:08.560520 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.560482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjc2\" (UniqueName: \"kubernetes.io/projected/a6e3649c-b5e4-451d-936c-b47e8baaeb2e-kube-api-access-7pjc2\") pod \"kserve-controller-manager-d6f4b65b8-bgkb4\" (UID: \"a6e3649c-b5e4-451d-936c-b47e8baaeb2e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:08.560634 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:08:08.560574 2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 20 05:08:08.560667 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:08:08.560652 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6e3649c-b5e4-451d-936c-b47e8baaeb2e-cert podName:a6e3649c-b5e4-451d-936c-b47e8baaeb2e nodeName:}" failed. No retries permitted until 2026-07-20 05:08:09.060631412 +0000 UTC m=+518.330115097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a6e3649c-b5e4-451d-936c-b47e8baaeb2e-cert") pod "kserve-controller-manager-d6f4b65b8-bgkb4" (UID: "a6e3649c-b5e4-451d-936c-b47e8baaeb2e") : secret "kserve-webhook-server-cert" not found Jul 20 05:08:08.562907 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.562880 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33b52d2a-b130-4f0f-b80c-7f698cb71ec7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gnfcq\" (UID: \"33b52d2a-b130-4f0f-b80c-7f698cb71ec7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:08.569556 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.569533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjc2\" (UniqueName: \"kubernetes.io/projected/a6e3649c-b5e4-451d-936c-b47e8baaeb2e-kube-api-access-7pjc2\") pod \"kserve-controller-manager-d6f4b65b8-bgkb4\" (UID: \"a6e3649c-b5e4-451d-936c-b47e8baaeb2e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:08.569818 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.569794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvsc\" (UniqueName: \"kubernetes.io/projected/33b52d2a-b130-4f0f-b80c-7f698cb71ec7-kube-api-access-8jvsc\") pod \"llmisvc-controller-manager-777c6bd7d5-gnfcq\" (UID: \"33b52d2a-b130-4f0f-b80c-7f698cb71ec7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:08.700178 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.700076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:08.829813 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:08.829787 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq"] Jul 20 05:08:08.832345 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:08:08.832314 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod33b52d2a_b130_4f0f_b80c_7f698cb71ec7.slice/crio-f3e84285484d04b7771383fd9d9a1cadf4a9c2a9537f2a592493a28ab8bb1264 WatchSource:0}: Error finding container f3e84285484d04b7771383fd9d9a1cadf4a9c2a9537f2a592493a28ab8bb1264: Status 404 returned error can't find the container with id f3e84285484d04b7771383fd9d9a1cadf4a9c2a9537f2a592493a28ab8bb1264 Jul 20 05:08:09.063436 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.063397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e3649c-b5e4-451d-936c-b47e8baaeb2e-cert\") pod \"kserve-controller-manager-d6f4b65b8-bgkb4\" (UID: \"a6e3649c-b5e4-451d-936c-b47e8baaeb2e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:09.065823 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.065797 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6e3649c-b5e4-451d-936c-b47e8baaeb2e-cert\") pod \"kserve-controller-manager-d6f4b65b8-bgkb4\" (UID: \"a6e3649c-b5e4-451d-936c-b47e8baaeb2e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:09.257606 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.257570 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-bf9656866-lwmqw"] Jul 20 05:08:09.261873 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.261852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.264685 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.264654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c4c318-4559-4148-ae06-c2a400fbe6b3-tls-certs\") pod \"kube-auth-proxy-bf9656866-lwmqw\" (UID: \"87c4c318-4559-4148-ae06-c2a400fbe6b3\") " pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.264800 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.264698 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87c4c318-4559-4148-ae06-c2a400fbe6b3-tmp\") pod \"kube-auth-proxy-bf9656866-lwmqw\" (UID: \"87c4c318-4559-4148-ae06-c2a400fbe6b3\") " pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.264800 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.264725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wbhq\" (UniqueName: \"kubernetes.io/projected/87c4c318-4559-4148-ae06-c2a400fbe6b3-kube-api-access-9wbhq\") pod \"kube-auth-proxy-bf9656866-lwmqw\" (UID: \"87c4c318-4559-4148-ae06-c2a400fbe6b3\") " pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.266079 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.266038 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 20 05:08:09.267259 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.267237 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 20 05:08:09.267360 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.267263 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9tfgf\"" Jul 20 05:08:09.267403 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.267365 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 20 05:08:09.267403 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.267387 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 20 05:08:09.273377 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.273358 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:09.283417 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.283389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bf9656866-lwmqw"] Jul 20 05:08:09.365660 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.365630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c4c318-4559-4148-ae06-c2a400fbe6b3-tls-certs\") pod \"kube-auth-proxy-bf9656866-lwmqw\" (UID: \"87c4c318-4559-4148-ae06-c2a400fbe6b3\") " pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.366013 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.365667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87c4c318-4559-4148-ae06-c2a400fbe6b3-tmp\") pod \"kube-auth-proxy-bf9656866-lwmqw\" (UID: \"87c4c318-4559-4148-ae06-c2a400fbe6b3\") " pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.366013 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.365700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbhq\" (UniqueName: \"kubernetes.io/projected/87c4c318-4559-4148-ae06-c2a400fbe6b3-kube-api-access-9wbhq\") pod \"kube-auth-proxy-bf9656866-lwmqw\" (UID: \"87c4c318-4559-4148-ae06-c2a400fbe6b3\") " pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.368070 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.368033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87c4c318-4559-4148-ae06-c2a400fbe6b3-tmp\") pod \"kube-auth-proxy-bf9656866-lwmqw\" (UID: \"87c4c318-4559-4148-ae06-c2a400fbe6b3\") " pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.368280 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.368263 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87c4c318-4559-4148-ae06-c2a400fbe6b3-tls-certs\") pod \"kube-auth-proxy-bf9656866-lwmqw\" (UID: \"87c4c318-4559-4148-ae06-c2a400fbe6b3\") " pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.376360 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.376296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbhq\" (UniqueName: \"kubernetes.io/projected/87c4c318-4559-4148-ae06-c2a400fbe6b3-kube-api-access-9wbhq\") pod \"kube-auth-proxy-bf9656866-lwmqw\" (UID: \"87c4c318-4559-4148-ae06-c2a400fbe6b3\") " pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.413345 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.413261 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4"] Jul 20 05:08:09.415803 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:08:09.415765 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e3649c_b5e4_451d_936c_b47e8baaeb2e.slice/crio-b4506942740e9b82ca6616a57e3f2f4bf1b3c00b81eeda0f0189cb86e880ff84 WatchSource:0}: Error finding container b4506942740e9b82ca6616a57e3f2f4bf1b3c00b81eeda0f0189cb86e880ff84: Status 404 returned error can't find the container with id b4506942740e9b82ca6616a57e3f2f4bf1b3c00b81eeda0f0189cb86e880ff84 Jul 20 05:08:09.425649 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.425619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" event={"ID":"33b52d2a-b130-4f0f-b80c-7f698cb71ec7","Type":"ContainerStarted","Data":"f3e84285484d04b7771383fd9d9a1cadf4a9c2a9537f2a592493a28ab8bb1264"} Jul 20 05:08:09.426667 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.426642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" event={"ID":"a6e3649c-b5e4-451d-936c-b47e8baaeb2e","Type":"ContainerStarted","Data":"b4506942740e9b82ca6616a57e3f2f4bf1b3c00b81eeda0f0189cb86e880ff84"} Jul 20 05:08:09.573917 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.573812 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" Jul 20 05:08:09.978815 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:09.978785 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bf9656866-lwmqw"] Jul 20 05:08:10.041482 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:08:10.041447 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c4c318_4559_4148_ae06_c2a400fbe6b3.slice/crio-10b1ce5392863617034403a3bf8f2b78d8a94308c1037bc78d35b7e58c043804 WatchSource:0}: Error finding container 10b1ce5392863617034403a3bf8f2b78d8a94308c1037bc78d35b7e58c043804: Status 404 returned error can't find the container with id 10b1ce5392863617034403a3bf8f2b78d8a94308c1037bc78d35b7e58c043804 Jul 20 05:08:10.436920 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.436874 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" event={"ID":"87c4c318-4559-4148-ae06-c2a400fbe6b3","Type":"ContainerStarted","Data":"10b1ce5392863617034403a3bf8f2b78d8a94308c1037bc78d35b7e58c043804"} Jul 20 05:08:10.857878 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.857840 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59"] Jul 20 05:08:10.861900 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.861498 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.864088 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.864037 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 20 05:08:10.864655 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.864277 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mngtj\"" Jul 20 05:08:10.864655 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.864318 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 20 05:08:10.864655 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.864482 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 20 05:08:10.864655 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.864557 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 20 05:08:10.873709 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.873648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59"] Jul 20 05:08:10.876609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.876009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.876609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.876072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.876609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.876116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.876609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.876178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fppx5\" (UniqueName: \"kubernetes.io/projected/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-kube-api-access-fppx5\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.876609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.876217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.876609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.876240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.876609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.876276 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.976796 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.976757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.976963 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.976812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.976963 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.976846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.976963 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.976884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.976963 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.976941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fppx5\" (UniqueName: \"kubernetes.io/projected/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-kube-api-access-fppx5\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.977209 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.976976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.977209 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.977001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.977769 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.977743 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.980882 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.980832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.981966 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.981924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.982672 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.982580 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.983273 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.983190 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.986254 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.986215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:10.987491 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:10.987452 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fppx5\" (UniqueName: \"kubernetes.io/projected/b02c8f0d-0e3a-4755-9aff-17c46c29b47a-kube-api-access-fppx5\") pod \"istiod-openshift-gateway-94bb8fbfd-wxx59\" (UID: \"b02c8f0d-0e3a-4755-9aff-17c46c29b47a\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:11.179850 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:11.179766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:13.000572 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.000542 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59"] Jul 20 05:08:13.002275 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:08:13.002250 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02c8f0d_0e3a_4755_9aff_17c46c29b47a.slice/crio-40727ce8399e595e96b008676a05c2ee2106464a85132ce9b7a9e1b76c3b83a3 WatchSource:0}: Error finding container 40727ce8399e595e96b008676a05c2ee2106464a85132ce9b7a9e1b76c3b83a3: Status 404 returned error can't find the container with id 40727ce8399e595e96b008676a05c2ee2106464a85132ce9b7a9e1b76c3b83a3 Jul 20 05:08:13.454841 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.454809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" event={"ID":"b02c8f0d-0e3a-4755-9aff-17c46c29b47a","Type":"ContainerStarted","Data":"40727ce8399e595e96b008676a05c2ee2106464a85132ce9b7a9e1b76c3b83a3"} Jul 20 05:08:13.456261 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.456234 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" event={"ID":"a6e3649c-b5e4-451d-936c-b47e8baaeb2e","Type":"ContainerStarted","Data":"3f4757c9b2de5287a4fa8cd9922b4791268a5ebe8fea0911d1bc3fbd14f51b70"} Jul 20 05:08:13.456398 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.456315 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:08:13.457666 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.457642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" event={"ID":"87c4c318-4559-4148-ae06-c2a400fbe6b3","Type":"ContainerStarted","Data":"dd8a2d5e1f335315dbd821b5100d9fe44f0191e8a54e70aee7d7e850b7ea01ab"} Jul 20 05:08:13.459039 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.459019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" event={"ID":"33b52d2a-b130-4f0f-b80c-7f698cb71ec7","Type":"ContainerStarted","Data":"cd1ed6d0d3f97d0c90e68ac7e355a562a6ec27eade443eabb91b21673075dcdc"} Jul 20 05:08:13.459140 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.459107 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:13.474765 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.474714 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" podStartSLOduration=2.051825957 podStartE2EDuration="5.474700364s" totalImagesPullingTime="3.422874407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:08:13.473595803 +0000 UTC m=+522.743079487" watchObservedRunningTime="2026-07-20 05:08:13.474700364 +0000 UTC m=+522.744184052" Jul 20 05:08:13.491592 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.491535 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" podStartSLOduration=1.487111633 podStartE2EDuration="5.491521363s" totalImagesPullingTime="4.00440973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:08:13.490395257 +0000 UTC m=+522.759878941" watchObservedRunningTime="2026-07-20 05:08:13.491521363 +0000 UTC m=+522.761005048" Jul 20 05:08:13.508017 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:13.507964 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-bf9656866-lwmqw" podStartSLOduration=1.7048652359999998 podStartE2EDuration="4.507948806s" totalImagesPullingTime="2.80308357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:08:13.506887487 +0000 UTC m=+522.776371172" watchObservedRunningTime="2026-07-20 05:08:13.507948806 +0000 UTC m=+522.777432490" Jul 20 05:08:14.406326 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:14.406280 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-pqmcj" Jul 20 05:08:14.409070 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:14.409029 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vxr6x" Jul 20 05:08:15.486090 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:15.486023 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:08:15.486334 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:15.486134 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:08:16.481180 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:16.481145 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" event={"ID":"b02c8f0d-0e3a-4755-9aff-17c46c29b47a","Type":"ContainerStarted","Data":"7352eca3e6a69a7442b69085c0c342ae5ae26bcc59c51b50e4ea7ec2e87e5d97"} Jul 20 05:08:16.481742 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:16.481489 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:16.483578 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:16.483546 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-wxx59 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 05:08:16.483784 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:16.483759 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" podUID="b02c8f0d-0e3a-4755-9aff-17c46c29b47a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 05:08:16.508129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:16.508041 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" podStartSLOduration=4.026624511 podStartE2EDuration="6.508019993s" totalImagesPullingTime="2.481395482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:08:16.50569628 +0000 UTC m=+525.775179975" watchObservedRunningTime="2026-07-20 05:08:16.508019993 +0000 UTC m=+525.777503679" Jul 20 05:08:17.486497 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:17.486469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-wxx59" Jul 20 05:08:44.472954 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:44.472922 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gnfcq" Jul 20 05:08:44.476138 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:08:44.476106 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bgkb4" Jul 20 05:09:44.436279 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:09:44.436246 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9664bb7f6-pcvpq"] Jul 20 05:10:01.333323 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.333286 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2"] Jul 20 05:10:01.336482 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.336466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:01.339745 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.339723 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9dxbz\"" Jul 20 05:10:01.339866 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.339746 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 20 05:10:01.340239 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.340219 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 20 05:10:01.357874 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.357842 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2"] Jul 20 05:10:01.410802 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.410760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn22h\" (UniqueName: \"kubernetes.io/projected/c3d88513-58e6-4a48-a0e3-ff97fb501611-kube-api-access-rn22h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t7cq2\" (UID: \"c3d88513-58e6-4a48-a0e3-ff97fb501611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:01.410966 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.410867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3d88513-58e6-4a48-a0e3-ff97fb501611-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t7cq2\" (UID: \"c3d88513-58e6-4a48-a0e3-ff97fb501611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:01.511437 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.511397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn22h\" (UniqueName: \"kubernetes.io/projected/c3d88513-58e6-4a48-a0e3-ff97fb501611-kube-api-access-rn22h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t7cq2\" (UID: \"c3d88513-58e6-4a48-a0e3-ff97fb501611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:01.511611 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.511460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3d88513-58e6-4a48-a0e3-ff97fb501611-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t7cq2\" (UID: \"c3d88513-58e6-4a48-a0e3-ff97fb501611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:01.511813 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.511796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3d88513-58e6-4a48-a0e3-ff97fb501611-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t7cq2\" (UID: \"c3d88513-58e6-4a48-a0e3-ff97fb501611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:01.520021 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.519988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn22h\" (UniqueName: \"kubernetes.io/projected/c3d88513-58e6-4a48-a0e3-ff97fb501611-kube-api-access-rn22h\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t7cq2\" (UID: \"c3d88513-58e6-4a48-a0e3-ff97fb501611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:01.646867 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.646780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:01.796085 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.796036 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2"] Jul 20 05:10:01.799489 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:10:01.799464 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d88513_58e6_4a48_a0e3_ff97fb501611.slice/crio-0a2874a1c97c9c617489bbaf55f3dd5148e5cce77d93d904c48318f2857050e0 WatchSource:0}: Error finding container 0a2874a1c97c9c617489bbaf55f3dd5148e5cce77d93d904c48318f2857050e0: Status 404 returned error can't find the container with id 0a2874a1c97c9c617489bbaf55f3dd5148e5cce77d93d904c48318f2857050e0 Jul 20 05:10:01.949099 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:01.948984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" event={"ID":"c3d88513-58e6-4a48-a0e3-ff97fb501611","Type":"ContainerStarted","Data":"0a2874a1c97c9c617489bbaf55f3dd5148e5cce77d93d904c48318f2857050e0"} Jul 20 05:10:02.104692 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.104662 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2"] Jul 20 05:10:02.109854 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.109824 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2"] Jul 20 05:10:02.124009 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.123981 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp"] Jul 20 05:10:02.128408 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.128384 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:02.140886 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.140860 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp"] Jul 20 05:10:02.224446 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.224343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955be396-2338-44bd-8828-6bc4304e18ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vnhsp\" (UID: \"955be396-2338-44bd-8828-6bc4304e18ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:02.224446 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.224390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwbz8\" (UniqueName: \"kubernetes.io/projected/955be396-2338-44bd-8828-6bc4304e18ea-kube-api-access-mwbz8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vnhsp\" (UID: \"955be396-2338-44bd-8828-6bc4304e18ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:02.325348 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.325310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955be396-2338-44bd-8828-6bc4304e18ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vnhsp\" (UID: \"955be396-2338-44bd-8828-6bc4304e18ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:02.325516 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.325351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwbz8\" (UniqueName: \"kubernetes.io/projected/955be396-2338-44bd-8828-6bc4304e18ea-kube-api-access-mwbz8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vnhsp\" (UID: \"955be396-2338-44bd-8828-6bc4304e18ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:02.325796 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.325771 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955be396-2338-44bd-8828-6bc4304e18ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vnhsp\" (UID: \"955be396-2338-44bd-8828-6bc4304e18ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:02.333918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.333888 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwbz8\" (UniqueName: \"kubernetes.io/projected/955be396-2338-44bd-8828-6bc4304e18ea-kube-api-access-mwbz8\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-vnhsp\" (UID: \"955be396-2338-44bd-8828-6bc4304e18ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:02.440702 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.440670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:02.972733 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:02.972702 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp"] Jul 20 05:10:03.036798 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:10:03.036764 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955be396_2338_44bd_8828_6bc4304e18ea.slice/crio-b37565ccb64b295571e45de527a3d3552d61c79503440ce4b7d2cb7b77d73da6 WatchSource:0}: Error finding container b37565ccb64b295571e45de527a3d3552d61c79503440ce4b7d2cb7b77d73da6: Status 404 returned error can't find the container with id b37565ccb64b295571e45de527a3d3552d61c79503440ce4b7d2cb7b77d73da6 Jul 20 05:10:03.962938 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:03.962890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" event={"ID":"955be396-2338-44bd-8828-6bc4304e18ea","Type":"ContainerStarted","Data":"b37565ccb64b295571e45de527a3d3552d61c79503440ce4b7d2cb7b77d73da6"} Jul 20 05:10:06.978594 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:06.978535 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" podUID="c3d88513-58e6-4a48-a0e3-ff97fb501611" containerName="manager" containerID="cri-o://ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a" gracePeriod=2 Jul 20 05:10:06.980148 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:06.980118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" event={"ID":"955be396-2338-44bd-8828-6bc4304e18ea","Type":"ContainerStarted","Data":"6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f"} Jul 20 05:10:06.980275 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:06.980254 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:07.004231 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.004180 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" podStartSLOduration=2.113426411 podStartE2EDuration="5.004164046s" totalImagesPullingTime="2.890737635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:10:07.001388526 +0000 UTC m=+636.270872214" watchObservedRunningTime="2026-07-20 05:10:07.004164046 +0000 UTC m=+636.273647730" Jul 20 05:10:07.222041 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.222008 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:07.272204 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.272173 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3d88513-58e6-4a48-a0e3-ff97fb501611-extensions-socket-volume\") pod \"c3d88513-58e6-4a48-a0e3-ff97fb501611\" (UID: \"c3d88513-58e6-4a48-a0e3-ff97fb501611\") " Jul 20 05:10:07.272382 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.272225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn22h\" (UniqueName: \"kubernetes.io/projected/c3d88513-58e6-4a48-a0e3-ff97fb501611-kube-api-access-rn22h\") pod \"c3d88513-58e6-4a48-a0e3-ff97fb501611\" (UID: \"c3d88513-58e6-4a48-a0e3-ff97fb501611\") " Jul 20 05:10:07.272455 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.272430 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d88513-58e6-4a48-a0e3-ff97fb501611-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c3d88513-58e6-4a48-a0e3-ff97fb501611" (UID: "c3d88513-58e6-4a48-a0e3-ff97fb501611"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:10:07.272549 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.272536 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3d88513-58e6-4a48-a0e3-ff97fb501611-extensions-socket-volume\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:07.274352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.274328 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d88513-58e6-4a48-a0e3-ff97fb501611-kube-api-access-rn22h" (OuterVolumeSpecName: "kube-api-access-rn22h") pod "c3d88513-58e6-4a48-a0e3-ff97fb501611" (UID: "c3d88513-58e6-4a48-a0e3-ff97fb501611"). InnerVolumeSpecName "kube-api-access-rn22h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:10:07.373178 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.373127 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn22h\" (UniqueName: \"kubernetes.io/projected/c3d88513-58e6-4a48-a0e3-ff97fb501611-kube-api-access-rn22h\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:07.985679 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.985647 2579 generic.go:356] "Generic (PLEG): container finished" podID="c3d88513-58e6-4a48-a0e3-ff97fb501611" containerID="ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a" exitCode=2 Jul 20 05:10:07.986126 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.985693 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:07.986126 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.985745 2579 scope.go:117] "RemoveContainer" containerID="ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a" Jul 20 05:10:07.987977 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:10:07.987950 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-t7cq2\" is forbidden: User \"system:node:ip-10-0-133-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-196.ec2.internal' and this object" podUID="c3d88513-58e6-4a48-a0e3-ff97fb501611" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:07.994898 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.994878 2579 scope.go:117] "RemoveContainer" containerID="ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a" Jul 20 05:10:07.995157 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:10:07.995139 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a\": container with ID starting with ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a not found: ID does not exist" containerID="ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a" Jul 20 05:10:07.995215 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:07.995166 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a"} err="failed to get container status \"ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a\": rpc error: code = NotFound desc = could not find container \"ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a\": container with ID starting with ba13ebefc7cddfe53771b0a8df702822e9dbb38970ef6e5850f0ffce4f23ef1a not found: ID does not exist" Jul 20 05:10:07.996026 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:10:07.996001 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-t7cq2\" is forbidden: User \"system:node:ip-10-0-133-196.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-196.ec2.internal' and this object" podUID="c3d88513-58e6-4a48-a0e3-ff97fb501611" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t7cq2" Jul 20 05:10:09.154973 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.154939 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3d88513-58e6-4a48-a0e3-ff97fb501611" path="/var/lib/kubelet/pods/c3d88513-58e6-4a48-a0e3-ff97fb501611/volumes" Jul 20 05:10:09.457720 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.457610 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9664bb7f6-pcvpq" podUID="6dbf11ce-1264-429c-af83-ba48f946ad7e" containerName="console" containerID="cri-o://050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80" gracePeriod=15 Jul 20 05:10:09.707825 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.707765 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:10:09.795506 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.795470 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-oauth-config\") pod \"6dbf11ce-1264-429c-af83-ba48f946ad7e\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " Jul 20 05:10:09.795692 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.795513 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-config\") pod \"6dbf11ce-1264-429c-af83-ba48f946ad7e\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " Jul 20 05:10:09.795692 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.795544 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-oauth-serving-cert\") pod \"6dbf11ce-1264-429c-af83-ba48f946ad7e\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " Jul 20 05:10:09.795692 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.795569 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-trusted-ca-bundle\") pod \"6dbf11ce-1264-429c-af83-ba48f946ad7e\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " Jul 20 05:10:09.795692 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.795592 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-serving-cert\") pod \"6dbf11ce-1264-429c-af83-ba48f946ad7e\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " Jul 20 05:10:09.795692 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.795640 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhnwp\" (UniqueName: \"kubernetes.io/projected/6dbf11ce-1264-429c-af83-ba48f946ad7e-kube-api-access-jhnwp\") pod \"6dbf11ce-1264-429c-af83-ba48f946ad7e\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " Jul 20 05:10:09.795692 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.795681 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-service-ca\") pod \"6dbf11ce-1264-429c-af83-ba48f946ad7e\" (UID: \"6dbf11ce-1264-429c-af83-ba48f946ad7e\") " Jul 20 05:10:09.796186 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.796159 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-config" (OuterVolumeSpecName: "console-config") pod "6dbf11ce-1264-429c-af83-ba48f946ad7e" (UID: "6dbf11ce-1264-429c-af83-ba48f946ad7e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:10:09.796186 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.796175 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6dbf11ce-1264-429c-af83-ba48f946ad7e" (UID: "6dbf11ce-1264-429c-af83-ba48f946ad7e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:10:09.796354 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.796185 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6dbf11ce-1264-429c-af83-ba48f946ad7e" (UID: "6dbf11ce-1264-429c-af83-ba48f946ad7e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:10:09.796354 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.796317 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-service-ca" (OuterVolumeSpecName: "service-ca") pod "6dbf11ce-1264-429c-af83-ba48f946ad7e" (UID: "6dbf11ce-1264-429c-af83-ba48f946ad7e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:10:09.797750 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.797730 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6dbf11ce-1264-429c-af83-ba48f946ad7e" (UID: "6dbf11ce-1264-429c-af83-ba48f946ad7e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 05:10:09.798189 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.798164 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6dbf11ce-1264-429c-af83-ba48f946ad7e" (UID: "6dbf11ce-1264-429c-af83-ba48f946ad7e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 05:10:09.798274 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.798199 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dbf11ce-1264-429c-af83-ba48f946ad7e-kube-api-access-jhnwp" (OuterVolumeSpecName: "kube-api-access-jhnwp") pod "6dbf11ce-1264-429c-af83-ba48f946ad7e" (UID: "6dbf11ce-1264-429c-af83-ba48f946ad7e"). InnerVolumeSpecName "kube-api-access-jhnwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:10:09.896572 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.896531 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-oauth-config\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:09.896572 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.896568 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-config\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:09.896572 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.896578 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-oauth-serving-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:09.896797 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.896587 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-trusted-ca-bundle\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:09.896797 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.896596 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6dbf11ce-1264-429c-af83-ba48f946ad7e-console-serving-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:09.896797 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.896605 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhnwp\" (UniqueName: \"kubernetes.io/projected/6dbf11ce-1264-429c-af83-ba48f946ad7e-kube-api-access-jhnwp\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:09.896797 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.896613 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbf11ce-1264-429c-af83-ba48f946ad7e-service-ca\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:09.996889 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.996801 2579 generic.go:356] "Generic (PLEG): container finished" podID="6dbf11ce-1264-429c-af83-ba48f946ad7e" containerID="050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80" exitCode=0 Jul 20 05:10:09.996889 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.996858 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9664bb7f6-pcvpq" Jul 20 05:10:09.997128 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.996887 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9664bb7f6-pcvpq" event={"ID":"6dbf11ce-1264-429c-af83-ba48f946ad7e","Type":"ContainerDied","Data":"050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80"} Jul 20 05:10:09.997128 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.996940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9664bb7f6-pcvpq" event={"ID":"6dbf11ce-1264-429c-af83-ba48f946ad7e","Type":"ContainerDied","Data":"8728ed2091517b78c887f7b5be408446ae76587093881dced9a5087d20510e9a"} Jul 20 05:10:09.997128 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:09.996961 2579 scope.go:117] "RemoveContainer" containerID="050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80" Jul 20 05:10:10.006611 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:10.006591 2579 scope.go:117] "RemoveContainer" containerID="050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80" Jul 20 05:10:10.006905 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:10:10.006879 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80\": container with ID starting with 050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80 not found: ID does not exist" containerID="050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80" Jul 20 05:10:10.006971 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:10.006908 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80"} err="failed to get container status \"050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80\": rpc error: code = NotFound desc = could not find container \"050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80\": container with ID starting with 050b75e17ba25c877b67ec9480bdb24d9c22a2d62ad99e086abacb486b7f4e80 not found: ID does not exist" Jul 20 05:10:10.017409 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:10.017356 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9664bb7f6-pcvpq"] Jul 20 05:10:10.019314 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:10.019290 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9664bb7f6-pcvpq"] Jul 20 05:10:11.154857 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:11.154822 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dbf11ce-1264-429c-af83-ba48f946ad7e" path="/var/lib/kubelet/pods/6dbf11ce-1264-429c-af83-ba48f946ad7e/volumes" Jul 20 05:10:17.988087 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:17.988035 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:18.218373 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.218336 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8"] Jul 20 05:10:18.218673 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.218662 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dbf11ce-1264-429c-af83-ba48f946ad7e" containerName="console" Jul 20 05:10:18.218728 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.218675 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dbf11ce-1264-429c-af83-ba48f946ad7e" containerName="console" Jul 20 05:10:18.218728 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.218684 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d88513-58e6-4a48-a0e3-ff97fb501611" containerName="manager" Jul 20 05:10:18.218728 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.218689 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d88513-58e6-4a48-a0e3-ff97fb501611" containerName="manager" Jul 20 05:10:18.218826 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.218746 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dbf11ce-1264-429c-af83-ba48f946ad7e" containerName="console" Jul 20 05:10:18.218826 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.218753 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d88513-58e6-4a48-a0e3-ff97fb501611" containerName="manager" Jul 20 05:10:18.223069 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.223042 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:18.233566 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.233541 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8"] Jul 20 05:10:18.366765 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.366737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-496m2\" (UniqueName: \"kubernetes.io/projected/bfd32a28-791a-445d-9f9e-d026e8d2b731-kube-api-access-496m2\") pod \"kuadrant-operator-controller-manager-86878c769f-ccrn8\" (UID: \"bfd32a28-791a-445d-9f9e-d026e8d2b731\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:18.366900 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.366774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd32a28-791a-445d-9f9e-d026e8d2b731-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ccrn8\" (UID: \"bfd32a28-791a-445d-9f9e-d026e8d2b731\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:18.468342 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.468293 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-496m2\" (UniqueName: \"kubernetes.io/projected/bfd32a28-791a-445d-9f9e-d026e8d2b731-kube-api-access-496m2\") pod \"kuadrant-operator-controller-manager-86878c769f-ccrn8\" (UID: \"bfd32a28-791a-445d-9f9e-d026e8d2b731\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:18.468519 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.468357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd32a28-791a-445d-9f9e-d026e8d2b731-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ccrn8\" (UID: \"bfd32a28-791a-445d-9f9e-d026e8d2b731\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:18.468864 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.468840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd32a28-791a-445d-9f9e-d026e8d2b731-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ccrn8\" (UID: \"bfd32a28-791a-445d-9f9e-d026e8d2b731\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:18.476807 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.476777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-496m2\" (UniqueName: \"kubernetes.io/projected/bfd32a28-791a-445d-9f9e-d026e8d2b731-kube-api-access-496m2\") pod \"kuadrant-operator-controller-manager-86878c769f-ccrn8\" (UID: \"bfd32a28-791a-445d-9f9e-d026e8d2b731\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:18.533529 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.533486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:18.667492 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:18.667458 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8"] Jul 20 05:10:18.670229 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:10:18.670195 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd32a28_791a_445d_9f9e_d026e8d2b731.slice/crio-32b7a97a1af1cc67786070e228125ee4bedd5c4d7b36378edef6455368ddb38b WatchSource:0}: Error finding container 32b7a97a1af1cc67786070e228125ee4bedd5c4d7b36378edef6455368ddb38b: Status 404 returned error can't find the container with id 32b7a97a1af1cc67786070e228125ee4bedd5c4d7b36378edef6455368ddb38b Jul 20 05:10:19.039834 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:19.039797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" event={"ID":"bfd32a28-791a-445d-9f9e-d026e8d2b731","Type":"ContainerStarted","Data":"33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a"} Jul 20 05:10:19.040334 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:19.039857 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" event={"ID":"bfd32a28-791a-445d-9f9e-d026e8d2b731","Type":"ContainerStarted","Data":"32b7a97a1af1cc67786070e228125ee4bedd5c4d7b36378edef6455368ddb38b"} Jul 20 05:10:19.040334 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:19.039925 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:19.058334 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:19.058280 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" podStartSLOduration=1.05826329 podStartE2EDuration="1.05826329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:10:19.056441698 +0000 UTC m=+648.325925400" watchObservedRunningTime="2026-07-20 05:10:19.05826329 +0000 UTC m=+648.327746976" Jul 20 05:10:30.046686 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.046650 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:10:30.099781 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.099747 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp"] Jul 20 05:10:30.099970 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.099948 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" podUID="955be396-2338-44bd-8828-6bc4304e18ea" containerName="manager" containerID="cri-o://6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f" gracePeriod=10 Jul 20 05:10:30.337474 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.337450 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:30.475212 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.475172 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955be396-2338-44bd-8828-6bc4304e18ea-extensions-socket-volume\") pod \"955be396-2338-44bd-8828-6bc4304e18ea\" (UID: \"955be396-2338-44bd-8828-6bc4304e18ea\") " Jul 20 05:10:30.475410 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.475259 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwbz8\" (UniqueName: \"kubernetes.io/projected/955be396-2338-44bd-8828-6bc4304e18ea-kube-api-access-mwbz8\") pod \"955be396-2338-44bd-8828-6bc4304e18ea\" (UID: \"955be396-2338-44bd-8828-6bc4304e18ea\") " Jul 20 05:10:30.475694 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.475667 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955be396-2338-44bd-8828-6bc4304e18ea-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "955be396-2338-44bd-8828-6bc4304e18ea" (UID: "955be396-2338-44bd-8828-6bc4304e18ea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:10:30.477509 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.477479 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955be396-2338-44bd-8828-6bc4304e18ea-kube-api-access-mwbz8" (OuterVolumeSpecName: "kube-api-access-mwbz8") pod "955be396-2338-44bd-8828-6bc4304e18ea" (UID: "955be396-2338-44bd-8828-6bc4304e18ea"). InnerVolumeSpecName "kube-api-access-mwbz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:10:30.576609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.576530 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955be396-2338-44bd-8828-6bc4304e18ea-extensions-socket-volume\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:30.576609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:30.576559 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwbz8\" (UniqueName: \"kubernetes.io/projected/955be396-2338-44bd-8828-6bc4304e18ea-kube-api-access-mwbz8\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:10:31.095020 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.094876 2579 generic.go:356] "Generic (PLEG): container finished" podID="955be396-2338-44bd-8828-6bc4304e18ea" containerID="6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f" exitCode=0 Jul 20 05:10:31.095020 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.094918 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" event={"ID":"955be396-2338-44bd-8828-6bc4304e18ea","Type":"ContainerDied","Data":"6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f"} Jul 20 05:10:31.095020 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.094940 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" Jul 20 05:10:31.095020 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.094983 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp" event={"ID":"955be396-2338-44bd-8828-6bc4304e18ea","Type":"ContainerDied","Data":"b37565ccb64b295571e45de527a3d3552d61c79503440ce4b7d2cb7b77d73da6"} Jul 20 05:10:31.095020 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.095013 2579 scope.go:117] "RemoveContainer" containerID="6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f" Jul 20 05:10:31.106102 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.106084 2579 scope.go:117] "RemoveContainer" containerID="6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f" Jul 20 05:10:31.106367 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:10:31.106348 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f\": container with ID starting with 6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f not found: ID does not exist" containerID="6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f" Jul 20 05:10:31.106428 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.106374 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f"} err="failed to get container status \"6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f\": rpc error: code = NotFound desc = could not find container \"6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f\": container with ID starting with 6248fdbb9afb84052fc41d65599058c9c9706740049ce3e21a157c7cb011676f not found: ID does not exist" Jul 20 05:10:31.120306 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.120270 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp"] Jul 20 05:10:31.124023 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.123996 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-vnhsp"] Jul 20 05:10:31.155571 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:31.155539 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="955be396-2338-44bd-8828-6bc4304e18ea" path="/var/lib/kubelet/pods/955be396-2338-44bd-8828-6bc4304e18ea/volumes" Jul 20 05:10:52.616471 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.616436 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:10:52.616939 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.616741 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955be396-2338-44bd-8828-6bc4304e18ea" containerName="manager" Jul 20 05:10:52.616939 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.616753 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="955be396-2338-44bd-8828-6bc4304e18ea" containerName="manager" Jul 20 05:10:52.616939 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.616825 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="955be396-2338-44bd-8828-6bc4304e18ea" containerName="manager" Jul 20 05:10:52.622013 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.621995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:10:52.624354 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.624326 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 20 05:10:52.624486 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.624465 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wx9pn\"" Jul 20 05:10:52.627836 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.627809 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:10:52.640602 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.640573 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:10:52.660695 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.660664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md62k\" (UniqueName: \"kubernetes.io/projected/12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8-kube-api-access-md62k\") pod \"limitador-limitador-54d6d988d5-dm8sm\" (UID: \"12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:10:52.660870 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.660718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8-config-file\") pod \"limitador-limitador-54d6d988d5-dm8sm\" (UID: \"12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:10:52.761574 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.761536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md62k\" (UniqueName: \"kubernetes.io/projected/12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8-kube-api-access-md62k\") pod \"limitador-limitador-54d6d988d5-dm8sm\" (UID: \"12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:10:52.761768 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.761592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8-config-file\") pod \"limitador-limitador-54d6d988d5-dm8sm\" (UID: \"12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:10:52.762209 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.762190 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8-config-file\") pod \"limitador-limitador-54d6d988d5-dm8sm\" (UID: \"12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:10:52.769887 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.769854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md62k\" (UniqueName: \"kubernetes.io/projected/12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8-kube-api-access-md62k\") pod \"limitador-limitador-54d6d988d5-dm8sm\" (UID: \"12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:10:52.934688 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:52.934592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:10:53.073565 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:53.073519 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:10:53.075741 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:10:53.075712 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ab696c_9f2b_44e9_a6c8_e6cd67bd74c8.slice/crio-3c45699c4cbb13ab9e4edd5b361d91f9687141eb2a7323754dde352e86721cd3 WatchSource:0}: Error finding container 3c45699c4cbb13ab9e4edd5b361d91f9687141eb2a7323754dde352e86721cd3: Status 404 returned error can't find the container with id 3c45699c4cbb13ab9e4edd5b361d91f9687141eb2a7323754dde352e86721cd3 Jul 20 05:10:53.195343 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:53.195251 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" event={"ID":"12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8","Type":"ContainerStarted","Data":"3c45699c4cbb13ab9e4edd5b361d91f9687141eb2a7323754dde352e86721cd3"} Jul 20 05:10:56.210717 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:56.210676 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" event={"ID":"12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8","Type":"ContainerStarted","Data":"af80f574dcfe4b8af9f6cba8089dd500b6dcd6416856960eebd9ebddd2768f59"} Jul 20 05:10:56.211194 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:56.210775 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:10:56.227096 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:10:56.227018 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" podStartSLOduration=1.754395103 podStartE2EDuration="4.227003325s" totalImagesPullingTime="2.472608222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:10:56.225096121 +0000 UTC m=+685.494579808" watchObservedRunningTime="2026-07-20 05:10:56.227003325 +0000 UTC m=+685.496487058" Jul 20 05:11:07.217210 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:11:07.217177 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dm8sm" Jul 20 05:12:09.307730 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:12:09.307696 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:12:32.024132 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:12:32.024081 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:12:47.718337 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:12:47.718291 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:13:00.431843 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:00.431764 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:13:07.314323 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:07.314286 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:13:31.922951 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.922908 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2"] Jul 20 05:13:31.926267 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.926245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:31.929187 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.929161 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 20 05:13:31.929322 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.929304 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 20 05:13:31.930385 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.930368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pg8cj\"" Jul 20 05:13:31.930441 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.930368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 20 05:13:31.935592 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.935568 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2"] Jul 20 05:13:31.989397 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.989362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc245fd5-991a-4d29-9918-8fe3f5ad18be-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:31.989565 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.989410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:31.989565 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.989435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:31.989663 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.989559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fntjg\" (UniqueName: \"kubernetes.io/projected/bc245fd5-991a-4d29-9918-8fe3f5ad18be-kube-api-access-fntjg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:31.989663 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.989613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:31.989663 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:31.989643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.090796 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.090755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.090796 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.090797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.091048 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.090861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fntjg\" (UniqueName: \"kubernetes.io/projected/bc245fd5-991a-4d29-9918-8fe3f5ad18be-kube-api-access-fntjg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.091048 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.090895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.091048 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.090920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.091048 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.090959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc245fd5-991a-4d29-9918-8fe3f5ad18be-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.091323 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.091301 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.091384 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.091350 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.091435 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.091378 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.093068 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.093044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc245fd5-991a-4d29-9918-8fe3f5ad18be-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.093523 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.093500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc245fd5-991a-4d29-9918-8fe3f5ad18be-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.098413 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.098392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fntjg\" (UniqueName: \"kubernetes.io/projected/bc245fd5-991a-4d29-9918-8fe3f5ad18be-kube-api-access-fntjg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-pxcr2\" (UID: \"bc245fd5-991a-4d29-9918-8fe3f5ad18be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.236846 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.236752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:32.369367 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.369340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2"] Jul 20 05:13:32.372308 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:13:32.372274 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc245fd5_991a_4d29_9918_8fe3f5ad18be.slice/crio-0f21c59c0c6806743fc482b1d92e12354b0a2f710dafcae832f49b2ecbcc62f2 WatchSource:0}: Error finding container 0f21c59c0c6806743fc482b1d92e12354b0a2f710dafcae832f49b2ecbcc62f2: Status 404 returned error can't find the container with id 0f21c59c0c6806743fc482b1d92e12354b0a2f710dafcae832f49b2ecbcc62f2 Jul 20 05:13:32.373721 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.373700 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 05:13:32.900632 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:32.900586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" event={"ID":"bc245fd5-991a-4d29-9918-8fe3f5ad18be","Type":"ContainerStarted","Data":"0f21c59c0c6806743fc482b1d92e12354b0a2f710dafcae832f49b2ecbcc62f2"} Jul 20 05:13:33.906351 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:33.906305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" event={"ID":"bc245fd5-991a-4d29-9918-8fe3f5ad18be","Type":"ContainerStarted","Data":"ee7e9b7228e695b1096f0dec124a6154d5b9d90839850384909e6be327687c79"} Jul 20 05:13:33.923543 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:33.923494 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" podStartSLOduration=1.904724575 podStartE2EDuration="2.923473808s" totalImagesPullingTime="1.018749233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:13:33.92222615 +0000 UTC m=+843.191709847" watchObservedRunningTime="2026-07-20 05:13:33.923473808 +0000 UTC m=+843.192957493" Jul 20 05:13:34.918086 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:34.918036 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:13:42.237575 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:42.237536 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:42.237966 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:42.237605 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:42.250258 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:42.250230 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:42.959084 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:42.959031 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-pxcr2" Jul 20 05:13:53.226196 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:13:53.226157 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:15:38.825671 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:15:38.825629 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:15:48.911743 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:15:48.911699 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:15:59.321678 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:15:59.321580 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:17:40.618117 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:17:40.618072 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:18:20.815022 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:18:20.814988 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:18:30.018765 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:18:30.018725 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:18:38.720294 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:18:38.720251 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:18:47.920804 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:18:47.920766 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:18:58.417748 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:18:58.417665 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:19:07.924099 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:19:07.924049 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:19:19.122201 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:19:19.122163 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:19:29.218281 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:19:29.218248 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:19:41.220220 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:19:41.220186 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:19:49.310594 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:19:49.310558 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:20:00.811189 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:20:00.811155 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:20:12.520510 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:20:12.520469 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:20:35.522958 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:20:35.522876 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:20:46.221753 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:20:46.221708 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:21:00.718462 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:21:00.718420 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:21:12.522870 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:21:12.522832 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:21:26.815691 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:21:26.815645 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:22:05.544540 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:05.544505 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8"] Jul 20 05:22:05.545175 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:05.544763 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" podUID="bfd32a28-791a-445d-9f9e-d026e8d2b731" containerName="manager" containerID="cri-o://33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a" gracePeriod=10 Jul 20 05:22:08.499223 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:08.499196 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:22:08.635386 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:08.635352 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd32a28-791a-445d-9f9e-d026e8d2b731-extensions-socket-volume\") pod \"bfd32a28-791a-445d-9f9e-d026e8d2b731\" (UID: \"bfd32a28-791a-445d-9f9e-d026e8d2b731\") " Jul 20 05:22:08.635549 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:08.635444 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-496m2\" (UniqueName: \"kubernetes.io/projected/bfd32a28-791a-445d-9f9e-d026e8d2b731-kube-api-access-496m2\") pod \"bfd32a28-791a-445d-9f9e-d026e8d2b731\" (UID: \"bfd32a28-791a-445d-9f9e-d026e8d2b731\") " Jul 20 05:22:08.635732 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:08.635711 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd32a28-791a-445d-9f9e-d026e8d2b731-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bfd32a28-791a-445d-9f9e-d026e8d2b731" (UID: "bfd32a28-791a-445d-9f9e-d026e8d2b731"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:22:08.637568 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:08.637540 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd32a28-791a-445d-9f9e-d026e8d2b731-kube-api-access-496m2" (OuterVolumeSpecName: "kube-api-access-496m2") pod "bfd32a28-791a-445d-9f9e-d026e8d2b731" (UID: "bfd32a28-791a-445d-9f9e-d026e8d2b731"). InnerVolumeSpecName "kube-api-access-496m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:22:08.737010 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:08.736966 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfd32a28-791a-445d-9f9e-d026e8d2b731-extensions-socket-volume\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:22:08.737010 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:08.737005 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-496m2\" (UniqueName: \"kubernetes.io/projected/bfd32a28-791a-445d-9f9e-d026e8d2b731-kube-api-access-496m2\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:22:09.175242 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:09.175208 2579 generic.go:356] "Generic (PLEG): container finished" podID="bfd32a28-791a-445d-9f9e-d026e8d2b731" containerID="33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a" exitCode=0 Jul 20 05:22:09.175381 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:09.175256 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" Jul 20 05:22:09.175381 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:09.175252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" event={"ID":"bfd32a28-791a-445d-9f9e-d026e8d2b731","Type":"ContainerDied","Data":"33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a"} Jul 20 05:22:09.175462 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:09.175380 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8" event={"ID":"bfd32a28-791a-445d-9f9e-d026e8d2b731","Type":"ContainerDied","Data":"32b7a97a1af1cc67786070e228125ee4bedd5c4d7b36378edef6455368ddb38b"} Jul 20 05:22:09.175462 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:09.175412 2579 scope.go:117] "RemoveContainer" containerID="33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a" Jul 20 05:22:09.185127 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:09.185106 2579 scope.go:117] "RemoveContainer" containerID="33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a" Jul 20 05:22:09.185379 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:22:09.185359 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a\": container with ID starting with 33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a not found: ID does not exist" containerID="33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a" Jul 20 05:22:09.185453 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:09.185392 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a"} err="failed to get container status \"33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a\": rpc error: code = NotFound desc = could not find container \"33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a\": container with ID starting with 33001a4aa22566a256b99261da0f40ced575e6629c10dc60b65d4d79a229380a not found: ID does not exist" Jul 20 05:22:09.193052 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:09.193026 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8"] Jul 20 05:22:09.198305 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:09.198284 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccrn8"] Jul 20 05:22:11.155947 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:22:11.155911 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd32a28-791a-445d-9f9e-d026e8d2b731" path="/var/lib/kubelet/pods/bfd32a28-791a-445d-9f9e-d026e8d2b731/volumes" Jul 20 05:23:06.684906 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.684867 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2"] Jul 20 05:23:06.685404 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.685193 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd32a28-791a-445d-9f9e-d026e8d2b731" containerName="manager" Jul 20 05:23:06.685404 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.685205 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd32a28-791a-445d-9f9e-d026e8d2b731" containerName="manager" Jul 20 05:23:06.685404 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.685268 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd32a28-791a-445d-9f9e-d026e8d2b731" containerName="manager" Jul 20 05:23:06.688194 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.688166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:06.691430 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.691413 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9dxbz\"" Jul 20 05:23:06.700424 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.700402 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2"] Jul 20 05:23:06.806152 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.806108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/732b8b95-40f0-472f-aada-5e5e16fddb82-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5bsx2\" (UID: \"732b8b95-40f0-472f-aada-5e5e16fddb82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:06.806343 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.806228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b54hq\" (UniqueName: \"kubernetes.io/projected/732b8b95-40f0-472f-aada-5e5e16fddb82-kube-api-access-b54hq\") pod \"kuadrant-operator-controller-manager-86878c769f-5bsx2\" (UID: \"732b8b95-40f0-472f-aada-5e5e16fddb82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:06.906862 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.906827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b54hq\" (UniqueName: \"kubernetes.io/projected/732b8b95-40f0-472f-aada-5e5e16fddb82-kube-api-access-b54hq\") pod \"kuadrant-operator-controller-manager-86878c769f-5bsx2\" (UID: \"732b8b95-40f0-472f-aada-5e5e16fddb82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:06.907011 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.906887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/732b8b95-40f0-472f-aada-5e5e16fddb82-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5bsx2\" (UID: \"732b8b95-40f0-472f-aada-5e5e16fddb82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:06.907247 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.907225 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/732b8b95-40f0-472f-aada-5e5e16fddb82-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5bsx2\" (UID: \"732b8b95-40f0-472f-aada-5e5e16fddb82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:06.915349 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.915326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b54hq\" (UniqueName: \"kubernetes.io/projected/732b8b95-40f0-472f-aada-5e5e16fddb82-kube-api-access-b54hq\") pod \"kuadrant-operator-controller-manager-86878c769f-5bsx2\" (UID: \"732b8b95-40f0-472f-aada-5e5e16fddb82\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:06.998026 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:06.997957 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:07.131125 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:07.131100 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2"] Jul 20 05:23:07.133564 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:23:07.133538 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732b8b95_40f0_472f_aada_5e5e16fddb82.slice/crio-5e584ec03357e58aadd78ed08cc059f20c27e3e5f1e1ce639ea237695460863a WatchSource:0}: Error finding container 5e584ec03357e58aadd78ed08cc059f20c27e3e5f1e1ce639ea237695460863a: Status 404 returned error can't find the container with id 5e584ec03357e58aadd78ed08cc059f20c27e3e5f1e1ce639ea237695460863a Jul 20 05:23:07.434493 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:07.434455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" event={"ID":"732b8b95-40f0-472f-aada-5e5e16fddb82","Type":"ContainerStarted","Data":"1927b90935054a146fa56c38733defc506895b44ab9f4c68f178c0a6194d7772"} Jul 20 05:23:07.434669 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:07.434513 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" event={"ID":"732b8b95-40f0-472f-aada-5e5e16fddb82","Type":"ContainerStarted","Data":"5e584ec03357e58aadd78ed08cc059f20c27e3e5f1e1ce639ea237695460863a"} Jul 20 05:23:07.434669 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:07.434554 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:07.453777 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:07.453623 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" podStartSLOduration=1.453606122 podStartE2EDuration="1.453606122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:23:07.452447641 +0000 UTC m=+1416.721931326" watchObservedRunningTime="2026-07-20 05:23:07.453606122 +0000 UTC m=+1416.723089808" Jul 20 05:23:18.442556 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:18.442529 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5bsx2" Jul 20 05:23:28.124668 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:28.124586 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:23:37.019301 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:37.019260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:23:49.336814 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:23:49.336780 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:24:00.536510 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:24:00.536468 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:24:12.018095 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:24:12.018046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:24:22.229930 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:24:22.229895 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:24:32.124529 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:24:32.124495 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:24:42.715955 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:24:42.715912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:24:53.017983 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:24:53.017947 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:25:00.716594 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:25:00.716550 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:25:09.620425 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:25:09.620384 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:25:19.631760 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:25:19.631723 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:25:37.224922 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:25:37.224873 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:25:47.021993 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:25:47.021955 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:26:02.520389 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.520348 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v"] Jul 20 05:26:02.523696 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.523678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.525992 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.525970 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-203d8646-kserve-self-signed-certs\"" Jul 20 05:26:02.540900 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.540879 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v"] Jul 20 05:26:02.574799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.574773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-tmp-dir\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.574903 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.574819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-model-cache\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.574903 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.574871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8hx\" (UniqueName: \"kubernetes.io/projected/41f1bd30-bb57-4285-85f4-499ee9cb4984-kube-api-access-2r8hx\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.574903 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.574890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-home\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.575076 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.574906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-dshm\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.575076 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.574922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f1bd30-bb57-4285-85f4-499ee9cb4984-tls-certs\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.675325 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.675287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-tmp-dir\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.675545 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.675370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-model-cache\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.675545 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.675424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8hx\" (UniqueName: \"kubernetes.io/projected/41f1bd30-bb57-4285-85f4-499ee9cb4984-kube-api-access-2r8hx\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.675545 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.675455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-home\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.675545 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.675486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-dshm\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.675545 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.675512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f1bd30-bb57-4285-85f4-499ee9cb4984-tls-certs\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.676292 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.675730 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-tmp-dir\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.676292 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.675770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-model-cache\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.676292 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.676008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-home\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.678800 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.678769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-dshm\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.678944 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.678925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f1bd30-bb57-4285-85f4-499ee9cb4984-tls-certs\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.683517 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.683496 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8hx\" (UniqueName: \"kubernetes.io/projected/41f1bd30-bb57-4285-85f4-499ee9cb4984-kube-api-access-2r8hx\") pod \"e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.832762 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.832685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:02.967099 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:02.967069 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v"] Jul 20 05:26:02.969183 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:26:02.969144 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41f1bd30_bb57_4285_85f4_499ee9cb4984.slice/crio-7be35177696bc190d0c7ddeb1e5b2a0de662c7097b6bf286fbff8ccfa745c7f8 WatchSource:0}: Error finding container 7be35177696bc190d0c7ddeb1e5b2a0de662c7097b6bf286fbff8ccfa745c7f8: Status 404 returned error can't find the container with id 7be35177696bc190d0c7ddeb1e5b2a0de662c7097b6bf286fbff8ccfa745c7f8 Jul 20 05:26:03.212565 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:03.212484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" event={"ID":"41f1bd30-bb57-4285-85f4-499ee9cb4984","Type":"ContainerStarted","Data":"9d7c3871dbbb65ccad5c70f8b1f65e0546f0cec1e4d541ae4361a15528334be6"} Jul 20 05:26:03.212565 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:03.212521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" event={"ID":"41f1bd30-bb57-4285-85f4-499ee9cb4984","Type":"ContainerStarted","Data":"7be35177696bc190d0c7ddeb1e5b2a0de662c7097b6bf286fbff8ccfa745c7f8"} Jul 20 05:26:03.233601 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:03.233556 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" podStartSLOduration=1.233541914 podStartE2EDuration="1.233541914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:26:03.233197646 +0000 UTC m=+1592.502681332" watchObservedRunningTime="2026-07-20 05:26:03.233541914 +0000 UTC m=+1592.503025598" Jul 20 05:26:06.723951 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:06.723916 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:26:12.833098 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:12.832993 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:12.833098 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:12.833042 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:12.845188 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:12.845166 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:13.265359 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:13.265280 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:26:20.616257 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:20.616221 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:26:57.332437 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:26:57.332349 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v"] Jul 20 05:27:00.925416 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:00.925380 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:27:12.352836 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.352771 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" podUID="41f1bd30-bb57-4285-85f4-499ee9cb4984" containerName="main" containerID="cri-o://9d7c3871dbbb65ccad5c70f8b1f65e0546f0cec1e4d541ae4361a15528334be6" gracePeriod=45 Jul 20 05:27:12.530579 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.530551 2579 generic.go:356] "Generic (PLEG): container finished" podID="41f1bd30-bb57-4285-85f4-499ee9cb4984" containerID="9d7c3871dbbb65ccad5c70f8b1f65e0546f0cec1e4d541ae4361a15528334be6" exitCode=0 Jul 20 05:27:12.530705 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.530621 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" event={"ID":"41f1bd30-bb57-4285-85f4-499ee9cb4984","Type":"ContainerDied","Data":"9d7c3871dbbb65ccad5c70f8b1f65e0546f0cec1e4d541ae4361a15528334be6"} Jul 20 05:27:12.603477 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.603424 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:27:12.644374 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.644342 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-model-cache\") pod \"41f1bd30-bb57-4285-85f4-499ee9cb4984\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " Jul 20 05:27:12.644513 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.644386 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r8hx\" (UniqueName: \"kubernetes.io/projected/41f1bd30-bb57-4285-85f4-499ee9cb4984-kube-api-access-2r8hx\") pod \"41f1bd30-bb57-4285-85f4-499ee9cb4984\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " Jul 20 05:27:12.644513 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.644410 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-dshm\") pod \"41f1bd30-bb57-4285-85f4-499ee9cb4984\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " Jul 20 05:27:12.644513 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.644459 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f1bd30-bb57-4285-85f4-499ee9cb4984-tls-certs\") pod \"41f1bd30-bb57-4285-85f4-499ee9cb4984\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " Jul 20 05:27:12.644668 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.644623 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-model-cache" (OuterVolumeSpecName: "model-cache") pod "41f1bd30-bb57-4285-85f4-499ee9cb4984" (UID: "41f1bd30-bb57-4285-85f4-499ee9cb4984"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:27:12.646626 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.646594 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-dshm" (OuterVolumeSpecName: "dshm") pod "41f1bd30-bb57-4285-85f4-499ee9cb4984" (UID: "41f1bd30-bb57-4285-85f4-499ee9cb4984"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:27:12.646726 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.646627 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f1bd30-bb57-4285-85f4-499ee9cb4984-kube-api-access-2r8hx" (OuterVolumeSpecName: "kube-api-access-2r8hx") pod "41f1bd30-bb57-4285-85f4-499ee9cb4984" (UID: "41f1bd30-bb57-4285-85f4-499ee9cb4984"). InnerVolumeSpecName "kube-api-access-2r8hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:27:12.646726 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.646650 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f1bd30-bb57-4285-85f4-499ee9cb4984-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41f1bd30-bb57-4285-85f4-499ee9cb4984" (UID: "41f1bd30-bb57-4285-85f4-499ee9cb4984"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 05:27:12.745044 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.745012 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-tmp-dir\") pod \"41f1bd30-bb57-4285-85f4-499ee9cb4984\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " Jul 20 05:27:12.745253 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.745083 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-home\") pod \"41f1bd30-bb57-4285-85f4-499ee9cb4984\" (UID: \"41f1bd30-bb57-4285-85f4-499ee9cb4984\") " Jul 20 05:27:12.745295 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.745277 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f1bd30-bb57-4285-85f4-499ee9cb4984-tls-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:27:12.745326 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.745297 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-model-cache\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:27:12.745326 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.745312 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r8hx\" (UniqueName: \"kubernetes.io/projected/41f1bd30-bb57-4285-85f4-499ee9cb4984-kube-api-access-2r8hx\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:27:12.745326 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.745315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41f1bd30-bb57-4285-85f4-499ee9cb4984" (UID: "41f1bd30-bb57-4285-85f4-499ee9cb4984"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:27:12.745421 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.745326 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-dshm\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:27:12.745421 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.745335 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-home" (OuterVolumeSpecName: "home") pod "41f1bd30-bb57-4285-85f4-499ee9cb4984" (UID: "41f1bd30-bb57-4285-85f4-499ee9cb4984"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:27:12.845707 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.845673 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-tmp-dir\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:27:12.845707 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:12.845703 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f1bd30-bb57-4285-85f4-499ee9cb4984-home\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:27:13.117959 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:13.117917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:27:13.536209 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:13.536181 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" Jul 20 05:27:13.536209 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:13.536193 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v" event={"ID":"41f1bd30-bb57-4285-85f4-499ee9cb4984","Type":"ContainerDied","Data":"7be35177696bc190d0c7ddeb1e5b2a0de662c7097b6bf286fbff8ccfa745c7f8"} Jul 20 05:27:13.536671 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:13.536243 2579 scope.go:117] "RemoveContainer" containerID="9d7c3871dbbb65ccad5c70f8b1f65e0546f0cec1e4d541ae4361a15528334be6" Jul 20 05:27:13.551599 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:13.551570 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v"] Jul 20 05:27:13.557963 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:13.557941 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-203d8646-kserve-b7cdbdf4c-hw85v"] Jul 20 05:27:15.154972 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:15.154928 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f1bd30-bb57-4285-85f4-499ee9cb4984" path="/var/lib/kubelet/pods/41f1bd30-bb57-4285-85f4-499ee9cb4984/volumes" Jul 20 05:27:18.615549 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:18.615513 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:27:36.017299 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:36.017264 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:27:49.821204 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:49.821167 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:27:58.912519 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:27:58.912440 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:28:13.318927 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:28:13.318893 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:28:22.319169 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:28:22.319125 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:28:35.120493 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:28:35.120457 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:28:44.623734 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:28:44.623690 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:28:54.227835 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:28:54.227796 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:29:05.218727 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:29:05.218692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:29:16.420838 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:29:16.420801 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:29:22.820704 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:29:22.820668 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:29:38.725867 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:29:38.725832 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:29:50.814882 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:29:50.814852 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:30:04.120149 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:30:04.120112 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:30:12.819416 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:30:12.819378 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:30:22.525944 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:30:22.525907 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:30:31.317473 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:30:31.317438 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:30:40.426511 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:30:40.426473 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:30:51.315067 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:30:51.315016 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:31:09.571655 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.571624 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck"] Jul 20 05:31:09.572038 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.571939 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f1bd30-bb57-4285-85f4-499ee9cb4984" containerName="main" Jul 20 05:31:09.572038 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.571949 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f1bd30-bb57-4285-85f4-499ee9cb4984" containerName="main" Jul 20 05:31:09.572038 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.572003 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f1bd30-bb57-4285-85f4-499ee9cb4984" containerName="main" Jul 20 05:31:09.574816 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.574799 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" Jul 20 05:31:09.577183 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.577154 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 20 05:31:09.577302 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.577200 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w55gj\"" Jul 20 05:31:09.581426 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.581401 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck"] Jul 20 05:31:09.588853 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.588831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ef220-31cf-4b8a-99fe-d2dd40b2cf5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50b57cd2-4plck\" (UID: \"587ef220-31cf-4b8a-99fe-d2dd40b2cf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" Jul 20 05:31:09.689461 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.689429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ef220-31cf-4b8a-99fe-d2dd40b2cf5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50b57cd2-4plck\" (UID: \"587ef220-31cf-4b8a-99fe-d2dd40b2cf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" Jul 20 05:31:09.689994 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.689976 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ef220-31cf-4b8a-99fe-d2dd40b2cf5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50b57cd2-4plck\" (UID: \"587ef220-31cf-4b8a-99fe-d2dd40b2cf5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" Jul 20 05:31:09.887619 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:09.887544 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" Jul 20 05:31:10.220323 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:10.220300 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck"] Jul 20 05:31:10.222403 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:31:10.222373 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587ef220_31cf_4b8a_99fe_d2dd40b2cf5e.slice/crio-490143cdc2b270da2947b4eaca499814c952eb09bb3e89d1e11201d14d9f6bf9 WatchSource:0}: Error finding container 490143cdc2b270da2947b4eaca499814c952eb09bb3e89d1e11201d14d9f6bf9: Status 404 returned error can't find the container with id 490143cdc2b270da2947b4eaca499814c952eb09bb3e89d1e11201d14d9f6bf9 Jul 20 05:31:10.224210 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:10.224190 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 20 05:31:10.594668 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:10.594633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" event={"ID":"587ef220-31cf-4b8a-99fe-d2dd40b2cf5e","Type":"ContainerStarted","Data":"490143cdc2b270da2947b4eaca499814c952eb09bb3e89d1e11201d14d9f6bf9"} Jul 20 05:31:13.612232 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:13.612199 2579 generic.go:356] "Generic (PLEG): container finished" podID="587ef220-31cf-4b8a-99fe-d2dd40b2cf5e" containerID="8a8f7a1469229caf18e0b2b57383d14bf45316698c82851197481b533460bc51" exitCode=0 Jul 20 05:31:13.612232 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:13.612235 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" event={"ID":"587ef220-31cf-4b8a-99fe-d2dd40b2cf5e","Type":"ContainerDied","Data":"8a8f7a1469229caf18e0b2b57383d14bf45316698c82851197481b533460bc51"} Jul 20 05:31:14.742324 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:14.742299 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" Jul 20 05:31:14.831206 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:14.831176 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ef220-31cf-4b8a-99fe-d2dd40b2cf5e-maas-api-service-ca\") pod \"587ef220-31cf-4b8a-99fe-d2dd40b2cf5e\" (UID: \"587ef220-31cf-4b8a-99fe-d2dd40b2cf5e\") " Jul 20 05:31:14.831552 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:14.831528 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587ef220-31cf-4b8a-99fe-d2dd40b2cf5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "587ef220-31cf-4b8a-99fe-d2dd40b2cf5e" (UID: "587ef220-31cf-4b8a-99fe-d2dd40b2cf5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:31:14.932127 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:14.932033 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/587ef220-31cf-4b8a-99fe-d2dd40b2cf5e-maas-api-service-ca\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:31:15.622755 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:15.622727 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" Jul 20 05:31:15.622920 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:15.622726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck" event={"ID":"587ef220-31cf-4b8a-99fe-d2dd40b2cf5e","Type":"ContainerDied","Data":"490143cdc2b270da2947b4eaca499814c952eb09bb3e89d1e11201d14d9f6bf9"} Jul 20 05:31:15.622920 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:15.622862 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490143cdc2b270da2947b4eaca499814c952eb09bb3e89d1e11201d14d9f6bf9" Jul 20 05:31:21.829475 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:21.829432 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:31:24.647612 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:24.647576 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck"] Jul 20 05:31:24.649217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:24.649195 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50b57cd2-4plck"] Jul 20 05:31:25.155340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.155305 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="587ef220-31cf-4b8a-99fe-d2dd40b2cf5e" path="/var/lib/kubelet/pods/587ef220-31cf-4b8a-99fe-d2dd40b2cf5e/volumes" Jul 20 05:31:25.771358 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.771323 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk"] Jul 20 05:31:25.771836 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.771751 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587ef220-31cf-4b8a-99fe-d2dd40b2cf5e" containerName="revoke-keys" Jul 20 05:31:25.771836 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.771766 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="587ef220-31cf-4b8a-99fe-d2dd40b2cf5e" containerName="revoke-keys" Jul 20 05:31:25.771957 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.771867 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="587ef220-31cf-4b8a-99fe-d2dd40b2cf5e" containerName="revoke-keys" Jul 20 05:31:25.775189 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.775170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.778010 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.777993 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-be1ac6ef-gw-openshift-default-dockercfg-7zkmr\"" Jul 20 05:31:25.789392 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.789367 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk"] Jul 20 05:31:25.820200 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.820171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-socket\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.820329 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.820235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-certs\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.820329 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.820269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-envoy\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.820329 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.820292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-data\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.820329 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.820319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-credential-socket\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.820461 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.820358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3d11f5-add6-45ad-9333-9903cf140893-istio-podinfo\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.820461 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.820428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-istio-token\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.820528 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.820459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpvrq\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-kube-api-access-lpvrq\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.820528 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.820515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3d11f5-add6-45ad-9333-9903cf140893-istiod-ca-cert\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921214 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-certs\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921384 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-envoy\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921384 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-data\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921384 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-credential-socket\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921384 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3d11f5-add6-45ad-9333-9903cf140893-istio-podinfo\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921384 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-istio-token\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921384 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpvrq\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-kube-api-access-lpvrq\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921670 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3d11f5-add6-45ad-9333-9903cf140893-istiod-ca-cert\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921670 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-socket\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921670 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921641 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-certs\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921820 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-data\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921820 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-credential-socket\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.921898 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.921826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-socket\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.922274 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.922248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3d11f5-add6-45ad-9333-9903cf140893-istiod-ca-cert\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.923689 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.923669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-envoy\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.923902 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.923883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3d11f5-add6-45ad-9333-9903cf140893-istio-podinfo\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.929350 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.929329 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-istio-token\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:25.929754 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:25.929734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpvrq\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-kube-api-access-lpvrq\") pod \"e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:26.085930 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:26.085851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:26.222676 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:26.222645 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk"] Jul 20 05:31:26.225763 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:31:26.225734 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3d11f5_add6_45ad_9333_9903cf140893.slice/crio-bedec49c5a3eb096e731dccd2255965b77a5befebd9ff78a7d1008dfdd651d4d WatchSource:0}: Error finding container bedec49c5a3eb096e731dccd2255965b77a5befebd9ff78a7d1008dfdd651d4d: Status 404 returned error can't find the container with id bedec49c5a3eb096e731dccd2255965b77a5befebd9ff78a7d1008dfdd651d4d Jul 20 05:31:26.670233 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:26.670193 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" event={"ID":"0f3d11f5-add6-45ad-9333-9903cf140893","Type":"ContainerStarted","Data":"bedec49c5a3eb096e731dccd2255965b77a5befebd9ff78a7d1008dfdd651d4d"} Jul 20 05:31:28.901661 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:28.901620 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:31:28.901940 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:28.901695 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:31:28.901940 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:28.901731 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:31:29.685716 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:29.685676 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" event={"ID":"0f3d11f5-add6-45ad-9333-9903cf140893","Type":"ContainerStarted","Data":"93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e"} Jul 20 05:31:29.706023 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:29.705975 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" podStartSLOduration=2.032460771 podStartE2EDuration="4.705959508s" totalImagesPullingTime="2.673498737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:31:29.704097895 +0000 UTC m=+1918.973581582" watchObservedRunningTime="2026-07-20 05:31:29.705959508 +0000 UTC m=+1918.975443193" Jul 20 05:31:30.086775 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:30.086742 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:30.091009 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:30.090985 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:30.690454 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:30.690424 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:30.691287 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:30.691266 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:34.816799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:34.816765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:31:42.756847 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:42.756763 2579 generic.go:356] "Generic (PLEG): container finished" podID="0f3d11f5-add6-45ad-9333-9903cf140893" containerID="93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e" exitCode=0 Jul 20 05:31:42.757242 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:42.756838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" event={"ID":"0f3d11f5-add6-45ad-9333-9903cf140893","Type":"ContainerDied","Data":"93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e"} Jul 20 05:31:42.757242 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:42.757237 2579 scope.go:117] "RemoveContainer" containerID="93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e" Jul 20 05:31:42.758231 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:42.758190 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:31:42.758297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:42.758282 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:31:42.758332 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:42.758319 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:31:43.763817 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:43.763776 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" event={"ID":"0f3d11f5-add6-45ad-9333-9903cf140893","Type":"ContainerStarted","Data":"4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c"} Jul 20 05:31:44.086693 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:44.086647 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:44.088184 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:44.088150 2579 patch_prober.go:28] interesting pod/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 20 05:31:44.088329 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:44.088213 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 20 05:31:45.086543 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:45.086504 2579 patch_prober.go:28] interesting pod/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 20 05:31:45.087160 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:45.087128 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 20 05:31:45.691107 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:45.691048 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:46.086606 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:46.086569 2579 patch_prober.go:28] interesting pod/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 20 05:31:46.087217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:46.087183 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 20 05:31:47.087137 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:47.087086 2579 patch_prober.go:28] interesting pod/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 20 05:31:47.087631 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:47.087158 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 20 05:31:48.090480 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:48.090448 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:31:48.091333 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:31:48.091314 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:32:09.515400 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:09.515363 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:32:11.328138 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:11.328105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk"] Jul 20 05:32:11.328540 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:11.328317 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" containerID="cri-o://4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c" gracePeriod=30 Jul 20 05:32:12.198151 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.198098 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q"] Jul 20 05:32:12.201541 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.201520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.205199 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.205178 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-defc77b8-openshift-default-dockercfg-mn6wq\"" Jul 20 05:32:12.221226 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.221203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-socket\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.221346 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.221237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istiod-ca-cert\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.221346 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.221257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-podinfo\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.221346 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.221332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-envoy\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.221508 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.221435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-credential-socket\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.221565 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.221526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-certs\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.221609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.221559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-data\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.221668 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.221646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-token\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.221732 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.221715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-kube-api-access-8s4xk\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.227601 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.227577 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q"] Jul 20 05:32:12.323008 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.322957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-credential-socket\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323223 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-certs\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323223 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-data\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323223 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-token\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323223 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-kube-api-access-8s4xk\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323439 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-socket\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323439 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istiod-ca-cert\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323439 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-podinfo\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323439 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-envoy\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323628 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-certs\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323628 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-credential-socket\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323628 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-data\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.323955 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.323927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-socket\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.324138 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.324120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istiod-ca-cert\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.325676 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.325649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-envoy\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.325907 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.325883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-podinfo\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.331498 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.331477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-kube-api-access-8s4xk\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.331784 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.331506 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-token\") pod \"e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.512127 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.512019 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:12.649229 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.649205 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q"] Jul 20 05:32:12.651313 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:32:12.651280 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8d0ce3_b13b_4065_9ec6_a24c6aff23d0.slice/crio-cc091ab4159b34e8b350ca3e2a122b905f90d80e7d8d9ede4fa27b8a40f33d4a WatchSource:0}: Error finding container cc091ab4159b34e8b350ca3e2a122b905f90d80e7d8d9ede4fa27b8a40f33d4a: Status 404 returned error can't find the container with id cc091ab4159b34e8b350ca3e2a122b905f90d80e7d8d9ede4fa27b8a40f33d4a Jul 20 05:32:12.653364 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.653328 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:32:12.653458 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.653413 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:32:12.653458 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.653442 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:32:12.905705 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.905666 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" event={"ID":"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0","Type":"ContainerStarted","Data":"c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345"} Jul 20 05:32:12.905871 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.905739 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" event={"ID":"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0","Type":"ContainerStarted","Data":"cc091ab4159b34e8b350ca3e2a122b905f90d80e7d8d9ede4fa27b8a40f33d4a"} Jul 20 05:32:12.926897 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:12.926843 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" podStartSLOduration=0.926826942 podStartE2EDuration="926.826942ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:32:12.925335631 +0000 UTC m=+1962.194819318" watchObservedRunningTime="2026-07-20 05:32:12.926826942 +0000 UTC m=+1962.196310682" Jul 20 05:32:13.512684 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:13.512651 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:13.517078 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:13.517039 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:13.910278 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:13.910248 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:13.911252 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:13.911237 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:16.575758 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.575728 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:32:16.664446 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664373 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3d11f5-add6-45ad-9333-9903cf140893-istio-podinfo\") pod \"0f3d11f5-add6-45ad-9333-9903cf140893\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " Jul 20 05:32:16.664446 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664434 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-socket\") pod \"0f3d11f5-add6-45ad-9333-9903cf140893\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " Jul 20 05:32:16.664657 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664463 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-certs\") pod \"0f3d11f5-add6-45ad-9333-9903cf140893\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " Jul 20 05:32:16.664657 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664500 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-envoy\") pod \"0f3d11f5-add6-45ad-9333-9903cf140893\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " Jul 20 05:32:16.664657 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664552 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-credential-socket\") pod \"0f3d11f5-add6-45ad-9333-9903cf140893\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " Jul 20 05:32:16.664657 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664596 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpvrq\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-kube-api-access-lpvrq\") pod \"0f3d11f5-add6-45ad-9333-9903cf140893\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " Jul 20 05:32:16.664657 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664619 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-data\") pod \"0f3d11f5-add6-45ad-9333-9903cf140893\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " Jul 20 05:32:16.664657 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664642 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-istio-token\") pod \"0f3d11f5-add6-45ad-9333-9903cf140893\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " Jul 20 05:32:16.664958 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3d11f5-add6-45ad-9333-9903cf140893-istiod-ca-cert\") pod \"0f3d11f5-add6-45ad-9333-9903cf140893\" (UID: \"0f3d11f5-add6-45ad-9333-9903cf140893\") " Jul 20 05:32:16.664958 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664797 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f3d11f5-add6-45ad-9333-9903cf140893" (UID: "0f3d11f5-add6-45ad-9333-9903cf140893"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:32:16.664958 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.664835 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f3d11f5-add6-45ad-9333-9903cf140893" (UID: "0f3d11f5-add6-45ad-9333-9903cf140893"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:32:16.665151 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.665031 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f3d11f5-add6-45ad-9333-9903cf140893" (UID: "0f3d11f5-add6-45ad-9333-9903cf140893"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:32:16.665215 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.665144 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3d11f5-add6-45ad-9333-9903cf140893-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f3d11f5-add6-45ad-9333-9903cf140893" (UID: "0f3d11f5-add6-45ad-9333-9903cf140893"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:32:16.665849 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.665820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f3d11f5-add6-45ad-9333-9903cf140893" (UID: "0f3d11f5-add6-45ad-9333-9903cf140893"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:32:16.665997 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.665976 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:32:16.666088 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.666002 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:32:16.666088 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.666016 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:32:16.666088 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.666028 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:32:16.666088 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.666041 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3d11f5-add6-45ad-9333-9903cf140893-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:32:16.667256 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.667222 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f3d11f5-add6-45ad-9333-9903cf140893" (UID: "0f3d11f5-add6-45ad-9333-9903cf140893"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:32:16.667370 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.667265 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f3d11f5-add6-45ad-9333-9903cf140893-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f3d11f5-add6-45ad-9333-9903cf140893" (UID: "0f3d11f5-add6-45ad-9333-9903cf140893"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:32:16.667370 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.667289 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-kube-api-access-lpvrq" (OuterVolumeSpecName: "kube-api-access-lpvrq") pod "0f3d11f5-add6-45ad-9333-9903cf140893" (UID: "0f3d11f5-add6-45ad-9333-9903cf140893"). InnerVolumeSpecName "kube-api-access-lpvrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:32:16.667794 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.667766 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f3d11f5-add6-45ad-9333-9903cf140893" (UID: "0f3d11f5-add6-45ad-9333-9903cf140893"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:32:16.767249 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.767217 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpvrq\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-kube-api-access-lpvrq\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:32:16.767249 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.767249 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3d11f5-add6-45ad-9333-9903cf140893-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:32:16.767406 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.767263 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3d11f5-add6-45ad-9333-9903cf140893-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:32:16.767406 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.767275 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3d11f5-add6-45ad-9333-9903cf140893-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:32:16.925965 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.925880 2579 generic.go:356] "Generic (PLEG): container finished" podID="0f3d11f5-add6-45ad-9333-9903cf140893" containerID="4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c" exitCode=0 Jul 20 05:32:16.925965 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.925930 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" event={"ID":"0f3d11f5-add6-45ad-9333-9903cf140893","Type":"ContainerDied","Data":"4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c"} Jul 20 05:32:16.925965 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.925945 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" Jul 20 05:32:16.926281 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.925994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk" event={"ID":"0f3d11f5-add6-45ad-9333-9903cf140893","Type":"ContainerDied","Data":"bedec49c5a3eb096e731dccd2255965b77a5befebd9ff78a7d1008dfdd651d4d"} Jul 20 05:32:16.926281 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.926019 2579 scope.go:117] "RemoveContainer" containerID="4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c" Jul 20 05:32:16.935473 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.935455 2579 scope.go:117] "RemoveContainer" containerID="93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e" Jul 20 05:32:16.945900 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.945878 2579 scope.go:117] "RemoveContainer" containerID="4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c" Jul 20 05:32:16.946236 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:32:16.946207 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c\": container with ID starting with 4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c not found: ID does not exist" containerID="4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c" Jul 20 05:32:16.946363 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.946258 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c"} err="failed to get container status \"4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c\": rpc error: code = NotFound desc = could not find container \"4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c\": container with ID starting with 4f9cb9309d467d321d35812c173bcef4e7345c5b4544f8ab8edf67e2a02ab38c not found: ID does not exist" Jul 20 05:32:16.946363 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.946284 2579 scope.go:117] "RemoveContainer" containerID="93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e" Jul 20 05:32:16.946883 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:32:16.946861 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e\": container with ID starting with 93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e not found: ID does not exist" containerID="93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e" Jul 20 05:32:16.946971 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.946887 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e"} err="failed to get container status \"93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e\": rpc error: code = NotFound desc = could not find container \"93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e\": container with ID starting with 93c51e97d3f7f19a2ba757be3767aa94d3c2189afda56599f3e0af998200b51e not found: ID does not exist" Jul 20 05:32:16.948682 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.948659 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk"] Jul 20 05:32:16.950105 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:16.950082 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-be1ac6ef-gw-openshift-default-5f7965ffb7-g97fk"] Jul 20 05:32:17.155522 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:17.155486 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" path="/var/lib/kubelet/pods/0f3d11f5-add6-45ad-9333-9903cf140893/volumes" Jul 20 05:32:20.518645 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:20.518607 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:32:28.912128 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:28.912086 2579 patch_prober.go:28] interesting pod/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 20 05:32:28.912611 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:28.912152 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 20 05:32:28.982367 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:28.982335 2579 generic.go:356] "Generic (PLEG): container finished" podID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerID="c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345" exitCode=0 Jul 20 05:32:28.982513 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:28.982416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" event={"ID":"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0","Type":"ContainerDied","Data":"c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345"} Jul 20 05:32:28.982781 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:28.982761 2579 scope.go:117] "RemoveContainer" containerID="c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345" Jul 20 05:32:28.983741 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:28.983714 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:32:28.983797 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:28.983785 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:32:28.983835 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:28.983818 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:32:30.003141 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:30.003101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" event={"ID":"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0","Type":"ContainerStarted","Data":"44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf"} Jul 20 05:32:30.513212 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:30.513169 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:30.513414 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:30.513392 2579 patch_prober.go:28] interesting pod/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 20 05:32:30.513497 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:30.513464 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 20 05:32:31.513249 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:31.513215 2579 patch_prober.go:28] interesting pod/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 20 05:32:31.513619 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:31.513267 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 20 05:32:32.513410 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:32.513372 2579 patch_prober.go:28] interesting pod/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 20 05:32:32.513791 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:32.513430 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 20 05:32:33.512717 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:33.512677 2579 patch_prober.go:28] interesting pod/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 20 05:32:33.512895 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:33.512751 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 20 05:32:34.515979 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:34.515949 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:34.516408 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:34.516320 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:34.516788 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:34.516771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:32:53.621119 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:53.621075 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:32:56.335850 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:56.335819 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q"] Jul 20 05:32:56.336329 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:56.336176 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" containerID="cri-o://44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf" gracePeriod=30 Jul 20 05:32:57.100174 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.100144 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w"] Jul 20 05:32:57.100486 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.100465 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" Jul 20 05:32:57.100486 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.100479 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" Jul 20 05:32:57.100643 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.100496 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" Jul 20 05:32:57.100643 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.100502 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" Jul 20 05:32:57.100643 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.100555 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" Jul 20 05:32:57.100643 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.100565 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f3d11f5-add6-45ad-9333-9903cf140893" containerName="istio-proxy" Jul 20 05:32:57.103675 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.103657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.106643 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.106619 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3a653f83-openshift-default-dockercfg-sjrn2\"" Jul 20 05:32:57.114338 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.114317 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w"] Jul 20 05:32:57.127207 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.127182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istiod-ca-cert\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.127340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.127223 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-certs\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.127340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.127254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-socket\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.127340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.127292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-envoy\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.127340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.127316 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rj8\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-kube-api-access-m4rj8\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.127594 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.127369 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-data\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.127594 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.127396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-podinfo\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.127594 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.127442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-credential-socket\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.127594 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.127479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-token\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-envoy\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rj8\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-kube-api-access-m4rj8\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228370 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-data\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228370 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-podinfo\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228370 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-credential-socket\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228370 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-token\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228370 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istiod-ca-cert\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228595 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-certs\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228595 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-socket\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228940 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-certs\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.228940 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-credential-socket\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.230705 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.228982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-socket\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.230705 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.229539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-data\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.230705 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.229582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istiod-ca-cert\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.231646 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.231610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-podinfo\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.234119 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.234092 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-envoy\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.237479 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.237451 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-token\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.237592 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.237572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rj8\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-kube-api-access-m4rj8\") pod \"e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.415311 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.415222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:57.553973 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.553944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w"] Jul 20 05:32:57.555867 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:32:57.555831 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b016dd_9d8e_4a6b_add4_678b3b3c4d98.slice/crio-e4db44190d1d22c9340ff0e1faedc05284f7175596726fb147049923696ca90a WatchSource:0}: Error finding container e4db44190d1d22c9340ff0e1faedc05284f7175596726fb147049923696ca90a: Status 404 returned error can't find the container with id e4db44190d1d22c9340ff0e1faedc05284f7175596726fb147049923696ca90a Jul 20 05:32:57.557935 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.557899 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:32:57.558015 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.557965 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:32:57.558015 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:57.557994 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:32:58.131040 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:58.131002 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" event={"ID":"91b016dd-9d8e-4a6b-add4-678b3b3c4d98","Type":"ContainerStarted","Data":"3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b"} Jul 20 05:32:58.131213 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:58.131053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" event={"ID":"91b016dd-9d8e-4a6b-add4-678b3b3c4d98","Type":"ContainerStarted","Data":"e4db44190d1d22c9340ff0e1faedc05284f7175596726fb147049923696ca90a"} Jul 20 05:32:58.153028 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:58.152979 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" podStartSLOduration=1.152961909 podStartE2EDuration="1.152961909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:32:58.150726186 +0000 UTC m=+2007.420209872" watchObservedRunningTime="2026-07-20 05:32:58.152961909 +0000 UTC m=+2007.422445593" Jul 20 05:32:58.415841 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:58.415761 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:58.420260 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:58.420236 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:59.136137 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:59.136108 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:32:59.137037 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:32:59.137017 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:33:01.591829 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.591799 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:33:01.669601 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669505 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-socket\") pod \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " Jul 20 05:33:01.669601 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669545 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-credential-socket\") pod \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " Jul 20 05:33:01.669601 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669563 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-data\") pod \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " Jul 20 05:33:01.669601 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669590 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-envoy\") pod \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " Jul 20 05:33:01.669919 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669617 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istiod-ca-cert\") pod \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " Jul 20 05:33:01.669919 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669657 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-kube-api-access-8s4xk\") pod \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " Jul 20 05:33:01.669919 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669691 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-certs\") pod \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " Jul 20 05:33:01.669919 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669727 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-podinfo\") pod \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " Jul 20 05:33:01.669919 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-token\") pod \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\" (UID: \"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0\") " Jul 20 05:33:01.669919 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669849 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" (UID: "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:01.669919 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.669877 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" (UID: "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:01.670307 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.670094 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:01.670307 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.670106 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" (UID: "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:33:01.670307 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.670113 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:01.670461 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.670359 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" (UID: "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:01.670461 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.670434 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" (UID: "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:01.672166 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.672134 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-kube-api-access-8s4xk" (OuterVolumeSpecName: "kube-api-access-8s4xk") pod "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" (UID: "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0"). InnerVolumeSpecName "kube-api-access-8s4xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:33:01.672297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.672198 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" (UID: "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:01.672434 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.672418 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" (UID: "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:33:01.672491 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.672442 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" (UID: "7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:33:01.770650 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.770616 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s4xk\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-kube-api-access-8s4xk\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:01.770650 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.770642 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:01.770650 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.770651 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:01.770650 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.770660 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:01.770978 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.770668 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:01.770978 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.770676 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:01.770978 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:01.770683 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:02.151220 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.151189 2579 generic.go:356] "Generic (PLEG): container finished" podID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerID="44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf" exitCode=0 Jul 20 05:33:02.151220 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.151225 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" Jul 20 05:33:02.151441 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.151272 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" event={"ID":"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0","Type":"ContainerDied","Data":"44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf"} Jul 20 05:33:02.151441 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.151321 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q" event={"ID":"7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0","Type":"ContainerDied","Data":"cc091ab4159b34e8b350ca3e2a122b905f90d80e7d8d9ede4fa27b8a40f33d4a"} Jul 20 05:33:02.151441 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.151348 2579 scope.go:117] "RemoveContainer" containerID="44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf" Jul 20 05:33:02.160537 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.160518 2579 scope.go:117] "RemoveContainer" containerID="c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345" Jul 20 05:33:02.168319 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.168303 2579 scope.go:117] "RemoveContainer" containerID="44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf" Jul 20 05:33:02.168586 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:33:02.168566 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf\": container with ID starting with 44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf not found: ID does not exist" containerID="44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf" Jul 20 05:33:02.168647 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.168593 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf"} err="failed to get container status \"44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf\": rpc error: code = NotFound desc = could not find container \"44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf\": container with ID starting with 44b5850f08a9acb04d6904e8e40c13326afe507f531ae48e2482bb86c456edbf not found: ID does not exist" Jul 20 05:33:02.168647 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.168611 2579 scope.go:117] "RemoveContainer" containerID="c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345" Jul 20 05:33:02.168846 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:33:02.168827 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345\": container with ID starting with c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345 not found: ID does not exist" containerID="c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345" Jul 20 05:33:02.168893 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.168852 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345"} err="failed to get container status \"c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345\": rpc error: code = NotFound desc = could not find container \"c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345\": container with ID starting with c2923517e794760fc00f08ef5d1b6ec65cb89cf1644ced766f2d7dd1366c6345 not found: ID does not exist" Jul 20 05:33:02.172907 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.172885 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q"] Jul 20 05:33:02.176377 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.176356 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-defc77b8-openshift-default-5bb8b49699-tpj5q"] Jul 20 05:33:02.627005 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:02.626973 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:33:03.161509 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:03.161465 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" path="/var/lib/kubelet/pods/7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0/volumes" Jul 20 05:33:11.207389 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:11.207351 2579 generic.go:356] "Generic (PLEG): container finished" podID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerID="3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b" exitCode=0 Jul 20 05:33:11.207847 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:11.207419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" event={"ID":"91b016dd-9d8e-4a6b-add4-678b3b3c4d98","Type":"ContainerDied","Data":"3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b"} Jul 20 05:33:11.207847 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:11.207770 2579 scope.go:117] "RemoveContainer" containerID="3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b" Jul 20 05:33:11.208735 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:11.208706 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:33:11.208825 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:11.208790 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:33:11.208883 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:11.208863 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:33:12.213756 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:12.213723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" event={"ID":"91b016dd-9d8e-4a6b-add4-678b3b3c4d98","Type":"ContainerStarted","Data":"7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e"} Jul 20 05:33:12.415792 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:12.415755 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:33:12.416946 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:12.416917 2579 patch_prober.go:28] interesting pod/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 20 05:33:12.417118 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:12.416983 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 20 05:33:13.416094 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:13.416037 2579 patch_prober.go:28] interesting pod/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 20 05:33:13.416485 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:13.416115 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 20 05:33:14.136773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:14.136737 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:33:14.416335 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:14.416254 2579 patch_prober.go:28] interesting pod/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 20 05:33:14.416335 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:14.416319 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 20 05:33:15.416447 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:15.416410 2579 patch_prober.go:28] interesting pod/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 20 05:33:15.416829 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:15.416468 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 20 05:33:16.419197 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:16.419163 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:33:16.420101 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:16.420079 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:33:37.714339 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:37.714292 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:33:39.592753 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:39.592698 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w"] Jul 20 05:33:39.593168 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:39.592983 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" containerID="cri-o://7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e" gracePeriod=30 Jul 20 05:33:44.849556 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:44.849533 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:33:45.036795 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.036771 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istiod-ca-cert\") pod \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " Jul 20 05:33:45.036969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.036821 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-envoy\") pod \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " Jul 20 05:33:45.036969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.036867 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-certs\") pod \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " Jul 20 05:33:45.036969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.036882 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-data\") pod \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " Jul 20 05:33:45.036969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.036917 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-podinfo\") pod \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " Jul 20 05:33:45.036969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.036943 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-socket\") pod \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " Jul 20 05:33:45.036969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.036969 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4rj8\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-kube-api-access-m4rj8\") pod \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " Jul 20 05:33:45.037308 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.037020 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-token\") pod \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " Jul 20 05:33:45.037308 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.037049 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-credential-socket\") pod \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\" (UID: \"91b016dd-9d8e-4a6b-add4-678b3b3c4d98\") " Jul 20 05:33:45.037308 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.037271 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91b016dd-9d8e-4a6b-add4-678b3b3c4d98" (UID: "91b016dd-9d8e-4a6b-add4-678b3b3c4d98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:33:45.037308 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.037283 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91b016dd-9d8e-4a6b-add4-678b3b3c4d98" (UID: "91b016dd-9d8e-4a6b-add4-678b3b3c4d98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:45.037505 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.037473 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91b016dd-9d8e-4a6b-add4-678b3b3c4d98" (UID: "91b016dd-9d8e-4a6b-add4-678b3b3c4d98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:45.037562 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.037501 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91b016dd-9d8e-4a6b-add4-678b3b3c4d98" (UID: "91b016dd-9d8e-4a6b-add4-678b3b3c4d98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:45.037994 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.037963 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-data" (OuterVolumeSpecName: "istio-data") pod "91b016dd-9d8e-4a6b-add4-678b3b3c4d98" (UID: "91b016dd-9d8e-4a6b-add4-678b3b3c4d98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:45.039287 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.039243 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91b016dd-9d8e-4a6b-add4-678b3b3c4d98" (UID: "91b016dd-9d8e-4a6b-add4-678b3b3c4d98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:33:45.039387 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.039299 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91b016dd-9d8e-4a6b-add4-678b3b3c4d98" (UID: "91b016dd-9d8e-4a6b-add4-678b3b3c4d98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:33:45.039387 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.039307 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-token" (OuterVolumeSpecName: "istio-token") pod "91b016dd-9d8e-4a6b-add4-678b3b3c4d98" (UID: "91b016dd-9d8e-4a6b-add4-678b3b3c4d98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:33:45.039387 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.039318 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-kube-api-access-m4rj8" (OuterVolumeSpecName: "kube-api-access-m4rj8") pod "91b016dd-9d8e-4a6b-add4-678b3b3c4d98" (UID: "91b016dd-9d8e-4a6b-add4-678b3b3c4d98"). InnerVolumeSpecName "kube-api-access-m4rj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:33:45.137877 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.137846 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:45.137877 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.137874 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:45.137877 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.137884 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:45.138101 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.137893 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:45.138101 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.137902 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:45.138101 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.137915 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:45.138101 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.137924 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:45.138101 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.137932 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:45.138101 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.137940 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4rj8\" (UniqueName: \"kubernetes.io/projected/91b016dd-9d8e-4a6b-add4-678b3b3c4d98-kube-api-access-m4rj8\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:33:45.371427 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.371336 2579 generic.go:356] "Generic (PLEG): container finished" podID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerID="7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e" exitCode=0 Jul 20 05:33:45.371427 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.371386 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" Jul 20 05:33:45.371648 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.371423 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" event={"ID":"91b016dd-9d8e-4a6b-add4-678b3b3c4d98","Type":"ContainerDied","Data":"7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e"} Jul 20 05:33:45.371648 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.371486 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w" event={"ID":"91b016dd-9d8e-4a6b-add4-678b3b3c4d98","Type":"ContainerDied","Data":"e4db44190d1d22c9340ff0e1faedc05284f7175596726fb147049923696ca90a"} Jul 20 05:33:45.371648 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.371517 2579 scope.go:117] "RemoveContainer" containerID="7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e" Jul 20 05:33:45.382202 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.382076 2579 scope.go:117] "RemoveContainer" containerID="3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b" Jul 20 05:33:45.390874 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.390632 2579 scope.go:117] "RemoveContainer" containerID="7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e" Jul 20 05:33:45.391004 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:33:45.390975 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e\": container with ID starting with 7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e not found: ID does not exist" containerID="7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e" Jul 20 05:33:45.391095 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.391016 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e"} err="failed to get container status \"7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e\": rpc error: code = NotFound desc = could not find container \"7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e\": container with ID starting with 7afb10bc93eb96ddd711e29f13543b36423e7b7dae3316390ebf0bccc94ea72e not found: ID does not exist" Jul 20 05:33:45.391095 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.391040 2579 scope.go:117] "RemoveContainer" containerID="3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b" Jul 20 05:33:45.391412 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:33:45.391392 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b\": container with ID starting with 3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b not found: ID does not exist" containerID="3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b" Jul 20 05:33:45.391474 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.391417 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b"} err="failed to get container status \"3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b\": rpc error: code = NotFound desc = could not find container \"3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b\": container with ID starting with 3d492a2ca90cfa6b58ef6ec21549a2bd110e2d5ba3d2dea4e83d1072bc24c22b not found: ID does not exist" Jul 20 05:33:45.392858 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.392836 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w"] Jul 20 05:33:45.396725 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:45.396705 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3a653f83-openshift-default-7d9b87bf5f-jxz7w"] Jul 20 05:33:47.155328 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:33:47.155295 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" path="/var/lib/kubelet/pods/91b016dd-9d8e-4a6b-add4-678b3b3c4d98/volumes" Jul 20 05:35:04.919855 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:04.919824 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:35:15.317877 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:15.317847 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:35:16.268342 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268307 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf"] Jul 20 05:35:16.268693 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268676 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" Jul 20 05:35:16.268754 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268695 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" Jul 20 05:35:16.268754 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268705 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" Jul 20 05:35:16.268754 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268710 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" Jul 20 05:35:16.268754 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268725 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" Jul 20 05:35:16.268754 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268731 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" Jul 20 05:35:16.268905 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268792 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" Jul 20 05:35:16.268905 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268803 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" Jul 20 05:35:16.268905 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268808 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" Jul 20 05:35:16.268905 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.268815 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91b016dd-9d8e-4a6b-add4-678b3b3c4d98" containerName="istio-proxy" Jul 20 05:35:16.271780 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.271763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.275324 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.275300 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d01a7226-openshift-default-dockercfg-n9cz7\"" Jul 20 05:35:16.298401 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.298374 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf"] Jul 20 05:35:16.371412 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.371375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-podinfo\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.371412 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.371409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t686s\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-kube-api-access-t686s\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.371918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.371435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-credential-socket\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.371918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.371499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-token\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.371918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.371530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-socket\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.371918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.371549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-data\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.371918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.371598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-envoy\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.371918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.371623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-certs\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.371918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.371653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istiod-ca-cert\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472454 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-certs\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472638 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istiod-ca-cert\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472638 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-podinfo\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472638 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t686s\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-kube-api-access-t686s\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472638 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-credential-socket\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472638 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-token\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472638 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-socket\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472947 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-data\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472947 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-certs\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.472947 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.472920 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-credential-socket\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.473137 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.473016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-socket\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.473195 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.473169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-envoy\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.473318 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.473297 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istiod-ca-cert\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.473375 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.473331 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-data\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.475382 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.475359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-envoy\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.475533 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.475514 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-podinfo\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.481469 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.481448 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t686s\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-kube-api-access-t686s\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.481568 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.481471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-token\") pod \"e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.581427 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.581342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:16.719639 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.719610 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf"] Jul 20 05:35:16.721297 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:35:16.721264 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e9d693_5bb7_4ef9_b1b2_00bc1e8dfd39.slice/crio-afeb1c54194509cda4cf5dd89974dae99636162f5b3ea397d703345b3ff14df1 WatchSource:0}: Error finding container afeb1c54194509cda4cf5dd89974dae99636162f5b3ea397d703345b3ff14df1: Status 404 returned error can't find the container with id afeb1c54194509cda4cf5dd89974dae99636162f5b3ea397d703345b3ff14df1 Jul 20 05:35:16.723238 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.723202 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:35:16.723341 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.723272 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:35:16.723341 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.723305 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:35:16.773493 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:16.773453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" event={"ID":"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39","Type":"ContainerStarted","Data":"afeb1c54194509cda4cf5dd89974dae99636162f5b3ea397d703345b3ff14df1"} Jul 20 05:35:17.779934 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:17.779900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" event={"ID":"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39","Type":"ContainerStarted","Data":"8fadc2613a10102f219533f5a39890aad5b436b5a24bfec0f30af783d932ecc2"} Jul 20 05:35:17.799382 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:17.799333 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" podStartSLOduration=1.799318515 podStartE2EDuration="1.799318515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:35:17.79853919 +0000 UTC m=+2147.068022880" watchObservedRunningTime="2026-07-20 05:35:17.799318515 +0000 UTC m=+2147.068802200" Jul 20 05:35:18.582387 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:18.582361 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:18.587011 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:18.586985 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:18.783892 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:18.783864 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:18.784783 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:18.784765 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:25.438995 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:25.438909 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:35:32.857840 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:32.857799 2579 generic.go:356] "Generic (PLEG): container finished" podID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerID="8fadc2613a10102f219533f5a39890aad5b436b5a24bfec0f30af783d932ecc2" exitCode=0 Jul 20 05:35:32.858258 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:32.857877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" event={"ID":"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39","Type":"ContainerDied","Data":"8fadc2613a10102f219533f5a39890aad5b436b5a24bfec0f30af783d932ecc2"} Jul 20 05:35:32.858342 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:32.858320 2579 scope.go:117] "RemoveContainer" containerID="8fadc2613a10102f219533f5a39890aad5b436b5a24bfec0f30af783d932ecc2" Jul 20 05:35:32.859285 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:32.859256 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:35:32.859346 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:32.859324 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:35:32.859387 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:32.859367 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:35:33.784513 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:33.784465 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:33.864120 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:33.864039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" event={"ID":"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39","Type":"ContainerStarted","Data":"d6897590c81a67ce347789ed1c7fea1a3c594361fb5b0c16a0631f14b247e0ce"} Jul 20 05:35:34.582028 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:34.581981 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:34.583490 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:34.583458 2579 patch_prober.go:28] interesting pod/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 20 05:35:34.583641 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:34.583533 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 20 05:35:35.585946 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:35.585904 2579 patch_prober.go:28] interesting pod/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 20 05:35:35.586627 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:35.586580 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 20 05:35:36.581792 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:36.581755 2579 patch_prober.go:28] interesting pod/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 20 05:35:36.582223 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:36.582193 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 20 05:35:37.585962 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:37.585927 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:37.586413 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:37.586334 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:37.587049 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:37.587029 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:44.817376 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:44.817337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:35:52.795714 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:52.795678 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf"] Jul 20 05:35:52.796101 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:52.795923 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" containerID="cri-o://d6897590c81a67ce347789ed1c7fea1a3c594361fb5b0c16a0631f14b247e0ce" gracePeriod=30 Jul 20 05:35:57.988193 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:57.988158 2579 generic.go:356] "Generic (PLEG): container finished" podID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerID="d6897590c81a67ce347789ed1c7fea1a3c594361fb5b0c16a0631f14b247e0ce" exitCode=0 Jul 20 05:35:57.988666 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:57.988211 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" event={"ID":"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39","Type":"ContainerDied","Data":"d6897590c81a67ce347789ed1c7fea1a3c594361fb5b0c16a0631f14b247e0ce"} Jul 20 05:35:57.988666 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:57.988263 2579 scope.go:117] "RemoveContainer" containerID="8fadc2613a10102f219533f5a39890aad5b436b5a24bfec0f30af783d932ecc2" Jul 20 05:35:58.045730 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.045707 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:58.125178 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125093 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istiod-ca-cert\") pod \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " Jul 20 05:35:58.125178 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125149 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-envoy\") pod \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " Jul 20 05:35:58.125388 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125201 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-podinfo\") pod \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " Jul 20 05:35:58.125388 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-credential-socket\") pod \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " Jul 20 05:35:58.125388 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-data\") pod \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " Jul 20 05:35:58.125388 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125276 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-socket\") pod \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " Jul 20 05:35:58.125388 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125298 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-certs\") pod \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " Jul 20 05:35:58.125388 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-token\") pod \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " Jul 20 05:35:58.125670 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125392 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t686s\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-kube-api-access-t686s\") pod \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\" (UID: \"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39\") " Jul 20 05:35:58.125670 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125591 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" (UID: "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:35:58.125670 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125597 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" (UID: "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:35:58.125670 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125525 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" (UID: "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:35:58.125801 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.125732 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" (UID: "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:35:58.126262 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.126227 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-data" (OuterVolumeSpecName: "istio-data") pod "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" (UID: "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:35:58.127604 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.127582 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-token" (OuterVolumeSpecName: "istio-token") pod "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" (UID: "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:35:58.127727 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.127713 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-kube-api-access-t686s" (OuterVolumeSpecName: "kube-api-access-t686s") pod "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" (UID: "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39"). InnerVolumeSpecName "kube-api-access-t686s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:35:58.127894 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.127870 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" (UID: "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:35:58.127894 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.127876 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" (UID: "85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:35:58.225964 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.225932 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:35:58.225964 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.225959 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:35:58.225964 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.225970 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:35:58.226230 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.225979 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:35:58.226230 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.225989 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:35:58.226230 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.225997 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:35:58.226230 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.226005 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:35:58.226230 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.226013 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t686s\" (UniqueName: \"kubernetes.io/projected/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-kube-api-access-t686s\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:35:58.226230 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.226022 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:35:58.994205 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.994176 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" Jul 20 05:35:58.994618 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.994215 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf" event={"ID":"85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39","Type":"ContainerDied","Data":"afeb1c54194509cda4cf5dd89974dae99636162f5b3ea397d703345b3ff14df1"} Jul 20 05:35:58.994618 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:58.994271 2579 scope.go:117] "RemoveContainer" containerID="d6897590c81a67ce347789ed1c7fea1a3c594361fb5b0c16a0631f14b247e0ce" Jul 20 05:35:59.014433 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:59.014405 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf"] Jul 20 05:35:59.018028 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:59.018004 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d01a7226-openshift-default-668c88c65f-mhhdf"] Jul 20 05:35:59.155306 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:35:59.155273 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" path="/var/lib/kubelet/pods/85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39/volumes" Jul 20 05:36:12.445946 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.445913 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk"] Jul 20 05:36:12.446465 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.446282 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" Jul 20 05:36:12.446465 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.446297 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" Jul 20 05:36:12.446465 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.446311 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" Jul 20 05:36:12.446465 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.446317 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8d0ce3-b13b-4065-9ec6-a24c6aff23d0" containerName="istio-proxy" Jul 20 05:36:12.446465 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.446387 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" Jul 20 05:36:12.449369 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.449339 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.451817 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.451793 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-77b596d2-openshift-default-dockercfg-btv4w\"" Jul 20 05:36:12.458632 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.458609 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk"] Jul 20 05:36:12.549117 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.549084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-token\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.549297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.549147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-envoy\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.549297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.549178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-certs\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.549297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.549200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-credential-socket\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.549297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.549217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfc4c\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-kube-api-access-cfc4c\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.549297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.549258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-socket\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.549297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.549278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-data\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.549297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.549295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-podinfo\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.549538 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.549341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istiod-ca-cert\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.649867 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.649828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-token\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650048 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.649908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-envoy\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650048 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.649947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-certs\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650048 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.649985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-credential-socket\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650048 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfc4c\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-kube-api-access-cfc4c\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650048 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-socket\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650368 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-data\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650368 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-podinfo\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650368 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istiod-ca-cert\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650528 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-certs\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650528 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650486 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-credential-socket\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650632 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-data\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650632 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-socket\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.650899 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.650879 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istiod-ca-cert\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.652357 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.652335 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-envoy\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.652488 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.652470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-podinfo\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.657361 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.657341 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-token\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.657445 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.657401 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfc4c\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-kube-api-access-cfc4c\") pod \"e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.761105 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.761035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:12.894503 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.894479 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk"] Jul 20 05:36:12.896915 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:36:12.896883 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5a54b0_4895_4e30_87b5_7176c0c8d7bf.slice/crio-5c77e6a4429287a394b6aae51301a82cfd9f81e24648c2fb16421eb8cdd15ab3 WatchSource:0}: Error finding container 5c77e6a4429287a394b6aae51301a82cfd9f81e24648c2fb16421eb8cdd15ab3: Status 404 returned error can't find the container with id 5c77e6a4429287a394b6aae51301a82cfd9f81e24648c2fb16421eb8cdd15ab3 Jul 20 05:36:12.899171 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.899136 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:36:12.899283 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.899211 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:36:12.899283 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:12.899256 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:36:13.058962 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:13.058925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" event={"ID":"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf","Type":"ContainerStarted","Data":"56d41b039a0471286325bfb14034b11a7f89d89bb0640a0cccafa1a53d283f09"} Jul 20 05:36:13.059170 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:13.058988 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" event={"ID":"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf","Type":"ContainerStarted","Data":"5c77e6a4429287a394b6aae51301a82cfd9f81e24648c2fb16421eb8cdd15ab3"} Jul 20 05:36:13.082833 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:13.082778 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" podStartSLOduration=1.082762844 podStartE2EDuration="1.082762844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:36:13.081210888 +0000 UTC m=+2202.350694573" watchObservedRunningTime="2026-07-20 05:36:13.082762844 +0000 UTC m=+2202.352246529" Jul 20 05:36:13.761320 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:13.761283 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:13.766495 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:13.766464 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:14.063829 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:14.063787 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:14.065515 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:14.065493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:20.368629 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:20.368591 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:36:25.614073 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.614030 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx"] Jul 20 05:36:25.614606 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.614568 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" Jul 20 05:36:25.614606 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.614604 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" Jul 20 05:36:25.614726 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.614679 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e9d693-5bb7-4ef9-b1b2-00bc1e8dfd39" containerName="istio-proxy" Jul 20 05:36:25.619154 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.619130 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.621948 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.621919 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-77b596d2\"/\"kube-root-ca.crt\"" Jul 20 05:36:25.622087 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.621948 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-77b596d2\"/\"default-dockercfg-qqznr\"" Jul 20 05:36:25.622937 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.622920 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-77b596d2\"/\"openshift-service-ca.crt\"" Jul 20 05:36:25.623314 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.623294 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-77b596d2\"/\"e2e-lifecycle-model-77b596d2-kserve-self-signed-certs\"" Jul 20 05:36:25.628461 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.628429 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx"] Jul 20 05:36:25.763982 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.763946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-dshm\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.764187 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.763992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-tmp-dir\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.764187 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.764093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef337bba-2e26-4751-8c00-1893f26a6b3b-tls-certs\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.764187 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.764162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-home\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.764319 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.764193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppjcs\" (UniqueName: \"kubernetes.io/projected/ef337bba-2e26-4751-8c00-1893f26a6b3b-kube-api-access-ppjcs\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.764319 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.764306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-model-cache\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.865856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.865520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-model-cache\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.865856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.865613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-dshm\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.865856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.865644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-tmp-dir\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.865856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.865676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef337bba-2e26-4751-8c00-1893f26a6b3b-tls-certs\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.866379 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.866349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-home\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.866527 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.866507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppjcs\" (UniqueName: \"kubernetes.io/projected/ef337bba-2e26-4751-8c00-1893f26a6b3b-kube-api-access-ppjcs\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.866677 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.866617 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-home\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.866677 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.866408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-tmp-dir\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.866867 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.866688 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-model-cache\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.869551 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.869507 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef337bba-2e26-4751-8c00-1893f26a6b3b-tls-certs\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.870304 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.870284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-dshm\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.874045 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.874017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppjcs\" (UniqueName: \"kubernetes.io/projected/ef337bba-2e26-4751-8c00-1893f26a6b3b-kube-api-access-ppjcs\") pod \"e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:25.931250 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:25.931213 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:26.083257 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:26.083219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx"] Jul 20 05:36:26.087021 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:36:26.086992 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef337bba_2e26_4751_8c00_1893f26a6b3b.slice/crio-0d6dee8d4bd7055d6f7d6d211b85b19e91eb8011baa17deec1fcd7e0d2ee2ebc WatchSource:0}: Error finding container 0d6dee8d4bd7055d6f7d6d211b85b19e91eb8011baa17deec1fcd7e0d2ee2ebc: Status 404 returned error can't find the container with id 0d6dee8d4bd7055d6f7d6d211b85b19e91eb8011baa17deec1fcd7e0d2ee2ebc Jul 20 05:36:26.121006 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:26.120941 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" event={"ID":"ef337bba-2e26-4751-8c00-1893f26a6b3b","Type":"ContainerStarted","Data":"0d6dee8d4bd7055d6f7d6d211b85b19e91eb8011baa17deec1fcd7e0d2ee2ebc"} Jul 20 05:36:27.127535 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:27.127496 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" event={"ID":"ef337bba-2e26-4751-8c00-1893f26a6b3b","Type":"ContainerStarted","Data":"ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e"} Jul 20 05:36:27.145189 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:27.145136 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" podStartSLOduration=2.145119551 podStartE2EDuration="2.145119551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:36:27.143013008 +0000 UTC m=+2216.412496705" watchObservedRunningTime="2026-07-20 05:36:27.145119551 +0000 UTC m=+2216.414603236" Jul 20 05:36:28.133762 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:28.133686 2579 generic.go:356] "Generic (PLEG): container finished" podID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerID="56d41b039a0471286325bfb14034b11a7f89d89bb0640a0cccafa1a53d283f09" exitCode=0 Jul 20 05:36:28.134189 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:28.133756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" event={"ID":"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf","Type":"ContainerDied","Data":"56d41b039a0471286325bfb14034b11a7f89d89bb0640a0cccafa1a53d283f09"} Jul 20 05:36:28.134276 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:28.134230 2579 scope.go:117] "RemoveContainer" containerID="56d41b039a0471286325bfb14034b11a7f89d89bb0640a0cccafa1a53d283f09" Jul 20 05:36:28.135115 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:28.135087 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:36:28.135171 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:28.135152 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:36:28.135207 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:28.135182 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:36:29.063741 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:29.063704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:29.140845 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:29.140806 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" event={"ID":"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf","Type":"ContainerStarted","Data":"f364af4ef0879ecc67d5afc5bd56b85a9ca6b5afeb39aa7eff043f5fb340a147"} Jul 20 05:36:29.727467 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:29.727433 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:36:29.761805 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:29.761776 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:29.762813 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:29.762789 2579 patch_prober.go:28] interesting pod/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 20 05:36:29.762896 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:29.762833 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 20 05:36:30.762243 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:30.762209 2579 patch_prober.go:28] interesting pod/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 20 05:36:30.762606 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:30.762265 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 20 05:36:31.762182 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:31.762085 2579 patch_prober.go:28] interesting pod/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 20 05:36:31.762182 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:31.762147 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 20 05:36:32.761864 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:32.761819 2579 patch_prober.go:28] interesting pod/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 20 05:36:32.762110 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:32.761886 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 20 05:36:33.764979 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:33.764952 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:33.765399 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:33.765222 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:33.765863 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:33.765845 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:36:35.932396 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:35.932357 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:35.932888 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:35.932417 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:35.944708 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:35.944681 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:36.184237 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:36.184141 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:36:44.822545 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:36:44.822503 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:37:02.228267 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:02.228228 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:37:12.327086 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:12.327037 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:37:15.401607 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:15.401570 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx"] Jul 20 05:37:30.421979 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.421919 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" podUID="ef337bba-2e26-4751-8c00-1893f26a6b3b" containerName="main" containerID="cri-o://ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e" gracePeriod=45 Jul 20 05:37:30.675616 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.675558 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:37:30.824457 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824429 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-dshm\") pod \"ef337bba-2e26-4751-8c00-1893f26a6b3b\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " Jul 20 05:37:30.824457 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824463 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-tmp-dir\") pod \"ef337bba-2e26-4751-8c00-1893f26a6b3b\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " Jul 20 05:37:30.824735 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824486 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppjcs\" (UniqueName: \"kubernetes.io/projected/ef337bba-2e26-4751-8c00-1893f26a6b3b-kube-api-access-ppjcs\") pod \"ef337bba-2e26-4751-8c00-1893f26a6b3b\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " Jul 20 05:37:30.824735 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824505 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef337bba-2e26-4751-8c00-1893f26a6b3b-tls-certs\") pod \"ef337bba-2e26-4751-8c00-1893f26a6b3b\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " Jul 20 05:37:30.824735 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824551 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-home\") pod \"ef337bba-2e26-4751-8c00-1893f26a6b3b\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " Jul 20 05:37:30.824735 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824594 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-model-cache\") pod \"ef337bba-2e26-4751-8c00-1893f26a6b3b\" (UID: \"ef337bba-2e26-4751-8c00-1893f26a6b3b\") " Jul 20 05:37:30.824950 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef337bba-2e26-4751-8c00-1893f26a6b3b" (UID: "ef337bba-2e26-4751-8c00-1893f26a6b3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:37:30.824950 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824868 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-home" (OuterVolumeSpecName: "home") pod "ef337bba-2e26-4751-8c00-1893f26a6b3b" (UID: "ef337bba-2e26-4751-8c00-1893f26a6b3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:37:30.824950 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824890 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef337bba-2e26-4751-8c00-1893f26a6b3b" (UID: "ef337bba-2e26-4751-8c00-1893f26a6b3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:37:30.825114 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.824998 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-home\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:37:30.825114 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.825017 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-model-cache\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:37:30.825114 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.825033 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-tmp-dir\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:37:30.826734 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.826712 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef337bba-2e26-4751-8c00-1893f26a6b3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef337bba-2e26-4751-8c00-1893f26a6b3b" (UID: "ef337bba-2e26-4751-8c00-1893f26a6b3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 05:37:30.826949 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.826932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-dshm" (OuterVolumeSpecName: "dshm") pod "ef337bba-2e26-4751-8c00-1893f26a6b3b" (UID: "ef337bba-2e26-4751-8c00-1893f26a6b3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:37:30.827023 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.827001 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef337bba-2e26-4751-8c00-1893f26a6b3b-kube-api-access-ppjcs" (OuterVolumeSpecName: "kube-api-access-ppjcs") pod "ef337bba-2e26-4751-8c00-1893f26a6b3b" (UID: "ef337bba-2e26-4751-8c00-1893f26a6b3b"). InnerVolumeSpecName "kube-api-access-ppjcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:37:30.926499 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.926408 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef337bba-2e26-4751-8c00-1893f26a6b3b-dshm\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:37:30.926499 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.926441 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppjcs\" (UniqueName: \"kubernetes.io/projected/ef337bba-2e26-4751-8c00-1893f26a6b3b-kube-api-access-ppjcs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:37:30.926499 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:30.926455 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef337bba-2e26-4751-8c00-1893f26a6b3b-tls-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:37:31.336518 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.336487 2579 scope.go:117] "RemoveContainer" containerID="8a8f7a1469229caf18e0b2b57383d14bf45316698c82851197481b533460bc51" Jul 20 05:37:31.442427 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.442399 2579 generic.go:356] "Generic (PLEG): container finished" podID="ef337bba-2e26-4751-8c00-1893f26a6b3b" containerID="ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e" exitCode=0 Jul 20 05:37:31.442873 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.442439 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" Jul 20 05:37:31.442873 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.442485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" event={"ID":"ef337bba-2e26-4751-8c00-1893f26a6b3b","Type":"ContainerDied","Data":"ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e"} Jul 20 05:37:31.442873 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.442530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx" event={"ID":"ef337bba-2e26-4751-8c00-1893f26a6b3b","Type":"ContainerDied","Data":"0d6dee8d4bd7055d6f7d6d211b85b19e91eb8011baa17deec1fcd7e0d2ee2ebc"} Jul 20 05:37:31.442873 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.442560 2579 scope.go:117] "RemoveContainer" containerID="ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e" Jul 20 05:37:31.450636 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.450617 2579 scope.go:117] "RemoveContainer" containerID="ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e" Jul 20 05:37:31.450857 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:37:31.450840 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e\": container with ID starting with ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e not found: ID does not exist" containerID="ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e" Jul 20 05:37:31.450899 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.450863 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e"} err="failed to get container status \"ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e\": rpc error: code = NotFound desc = could not find container \"ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e\": container with ID starting with ce84ba2229d526d71f7fc252bd48bb68117ee6d9cf899ad20ddb522d1e20983e not found: ID does not exist" Jul 20 05:37:31.459470 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.459449 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx"] Jul 20 05:37:31.465604 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:31.465574 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-77b596d2/e2e-lifecycle-model-77b596d2-kserve-6d5cf95f6c-25bqx"] Jul 20 05:37:33.154734 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:33.154693 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef337bba-2e26-4751-8c00-1893f26a6b3b" path="/var/lib/kubelet/pods/ef337bba-2e26-4751-8c00-1893f26a6b3b/volumes" Jul 20 05:37:58.413105 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:58.413073 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk"] Jul 20 05:37:58.413669 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:37:58.413333 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" containerID="cri-o://f364af4ef0879ecc67d5afc5bd56b85a9ca6b5afeb39aa7eff043f5fb340a147" gracePeriod=30 Jul 20 05:38:02.422162 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:02.422122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:38:03.594319 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.594289 2579 generic.go:356] "Generic (PLEG): container finished" podID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerID="f364af4ef0879ecc67d5afc5bd56b85a9ca6b5afeb39aa7eff043f5fb340a147" exitCode=0 Jul 20 05:38:03.594699 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.594365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" event={"ID":"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf","Type":"ContainerDied","Data":"f364af4ef0879ecc67d5afc5bd56b85a9ca6b5afeb39aa7eff043f5fb340a147"} Jul 20 05:38:03.594699 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.594449 2579 scope.go:117] "RemoveContainer" containerID="56d41b039a0471286325bfb14034b11a7f89d89bb0640a0cccafa1a53d283f09" Jul 20 05:38:03.668183 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.668162 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:38:03.791299 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791262 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-socket\") pod \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " Jul 20 05:38:03.791488 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791315 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-credential-socket\") pod \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " Jul 20 05:38:03.791488 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791346 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-certs\") pod \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " Jul 20 05:38:03.791488 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791368 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfc4c\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-kube-api-access-cfc4c\") pod \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " Jul 20 05:38:03.791488 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791398 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-data\") pod \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " Jul 20 05:38:03.791488 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791431 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-envoy\") pod \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " Jul 20 05:38:03.791488 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791466 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-token\") pod \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " Jul 20 05:38:03.791856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791524 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-podinfo\") pod \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " Jul 20 05:38:03.791856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791544 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istiod-ca-cert\") pod \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\" (UID: \"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf\") " Jul 20 05:38:03.791856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791618 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" (UID: "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:38:03.791856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791776 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" (UID: "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:38:03.792130 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791935 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:38:03.792130 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.791957 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:38:03.792130 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.792044 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" (UID: "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:38:03.792387 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.792352 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" (UID: "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:38:03.792612 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.792586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" (UID: "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:38:03.794280 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.794247 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-kube-api-access-cfc4c" (OuterVolumeSpecName: "kube-api-access-cfc4c") pod "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" (UID: "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf"). InnerVolumeSpecName "kube-api-access-cfc4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:38:03.794280 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.794264 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" (UID: "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:38:03.794420 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.794319 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" (UID: "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:38:03.794420 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.794340 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" (UID: "7b5a54b0-4895-4e30-87b5-7176c0c8d7bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:38:03.893276 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.893244 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:38:03.893276 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.893271 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfc4c\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-kube-api-access-cfc4c\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:38:03.893276 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.893280 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:38:03.893503 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.893288 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:38:03.893503 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.893296 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:38:03.893503 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.893303 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:38:03.893503 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:03.893310 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:38:04.600040 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:04.600012 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" Jul 20 05:38:04.600481 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:04.600009 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk" event={"ID":"7b5a54b0-4895-4e30-87b5-7176c0c8d7bf","Type":"ContainerDied","Data":"5c77e6a4429287a394b6aae51301a82cfd9f81e24648c2fb16421eb8cdd15ab3"} Jul 20 05:38:04.600481 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:04.600159 2579 scope.go:117] "RemoveContainer" containerID="f364af4ef0879ecc67d5afc5bd56b85a9ca6b5afeb39aa7eff043f5fb340a147" Jul 20 05:38:04.621511 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:04.621481 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk"] Jul 20 05:38:04.624670 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:04.624648 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-77b596d2-openshift-default-84fcb88b8c-pwjgk"] Jul 20 05:38:05.155049 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:05.155018 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" path="/var/lib/kubelet/pods/7b5a54b0-4895-4e30-87b5-7176c0c8d7bf/volumes" Jul 20 05:38:11.724410 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:11.724378 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:38:53.823559 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.823523 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7"] Jul 20 05:38:53.824051 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.823993 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef337bba-2e26-4751-8c00-1893f26a6b3b" containerName="main" Jul 20 05:38:53.824051 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.824012 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef337bba-2e26-4751-8c00-1893f26a6b3b" containerName="main" Jul 20 05:38:53.824051 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.824036 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" Jul 20 05:38:53.824051 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.824044 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" Jul 20 05:38:53.824294 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.824076 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" Jul 20 05:38:53.824294 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.824085 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" Jul 20 05:38:53.824294 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.824179 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef337bba-2e26-4751-8c00-1893f26a6b3b" containerName="main" Jul 20 05:38:53.824294 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.824190 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" Jul 20 05:38:53.824294 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.824199 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5a54b0-4895-4e30-87b5-7176c0c8d7bf" containerName="istio-proxy" Jul 20 05:38:53.827424 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.827400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:53.829874 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.829850 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-192e48-openshift-default-dockercfg-42khk\"" Jul 20 05:38:53.838649 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.838627 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7"] Jul 20 05:38:53.928596 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.928550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26154e2-5728-43ba-b19f-791db2f435ea-istio-podinfo\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:53.928780 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.928604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-credential-socket\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:53.928780 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.928632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-envoy\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:53.928780 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.928723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755ln\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-kube-api-access-755ln\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:53.928780 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.928757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-istio-token\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:53.928954 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.928833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-socket\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:53.928954 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.928929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-certs\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:53.928954 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.928952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-data\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:53.929145 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:53.928972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26154e2-5728-43ba-b19f-791db2f435ea-istiod-ca-cert\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030076 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-socket\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030076 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-certs\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-data\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26154e2-5728-43ba-b19f-791db2f435ea-istiod-ca-cert\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26154e2-5728-43ba-b19f-791db2f435ea-istio-podinfo\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-credential-socket\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030340 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-envoy\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030597 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755ln\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-kube-api-access-755ln\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030597 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-istio-token\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030705 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-certs\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030705 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-credential-socket\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030705 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-data\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030858 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030793 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-socket\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.030858 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.030832 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26154e2-5728-43ba-b19f-791db2f435ea-istiod-ca-cert\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.032550 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.032529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-envoy\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.032811 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.032794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26154e2-5728-43ba-b19f-791db2f435ea-istio-podinfo\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.037876 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.037852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-istio-token\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.037969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.037925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-755ln\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-kube-api-access-755ln\") pod \"e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.138133 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.138026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:54.278372 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.278346 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7"] Jul 20 05:38:54.280281 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:38:54.280252 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26154e2_5728_43ba_b19f_791db2f435ea.slice/crio-b750c77c43c91c441d0ce9c6e90cb4e94922a5b21234d46206fb8169aa82b998 WatchSource:0}: Error finding container b750c77c43c91c441d0ce9c6e90cb4e94922a5b21234d46206fb8169aa82b998: Status 404 returned error can't find the container with id b750c77c43c91c441d0ce9c6e90cb4e94922a5b21234d46206fb8169aa82b998 Jul 20 05:38:54.282523 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.282486 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:38:54.282630 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.282570 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:38:54.282694 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.282635 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:38:54.828907 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.828869 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" event={"ID":"d26154e2-5728-43ba-b19f-791db2f435ea","Type":"ContainerStarted","Data":"c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532"} Jul 20 05:38:54.829313 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.828934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" event={"ID":"d26154e2-5728-43ba-b19f-791db2f435ea","Type":"ContainerStarted","Data":"b750c77c43c91c441d0ce9c6e90cb4e94922a5b21234d46206fb8169aa82b998"} Jul 20 05:38:54.849587 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:54.849541 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" podStartSLOduration=1.8495255720000001 podStartE2EDuration="1.849525572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:38:54.84859967 +0000 UTC m=+2364.118083355" watchObservedRunningTime="2026-07-20 05:38:54.849525572 +0000 UTC m=+2364.119009257" Jul 20 05:38:55.138441 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:55.138357 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:55.143339 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:55.143314 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:55.834035 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:55.834002 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:38:55.835094 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:38:55.835073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:39:01.986582 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:01.986549 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6"] Jul 20 05:39:01.990119 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:01.990095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:01.992989 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:01.992971 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-598bdb-openshift-default-dockercfg-nl87p\"" Jul 20 05:39:02.001485 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.001463 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6"] Jul 20 05:39:02.101464 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.101422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-podinfo\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.101755 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.101497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-credential-socket\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.101755 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.101541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-certs\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.101755 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.101602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-token\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.101755 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.101644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-data\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.101755 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.101686 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-envoy\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.101755 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.101717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-socket\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.101755 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.101749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istiod-ca-cert\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.102138 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.101788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqt8w\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-kube-api-access-vqt8w\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.202550 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-socket\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.202550 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istiod-ca-cert\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.202799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqt8w\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-kube-api-access-vqt8w\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.202799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-podinfo\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.202799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-credential-socket\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.202799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-certs\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.202799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-token\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.203082 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-data\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.203082 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-envoy\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.203082 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-socket\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.203082 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.202986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-credential-socket\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.203291 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.203087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-certs\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.203291 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.203245 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-data\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.203377 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.203355 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istiod-ca-cert\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.205187 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.205167 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-envoy\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.205444 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.205426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-podinfo\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.210698 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.210677 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-token\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.210851 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.210824 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqt8w\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-kube-api-access-vqt8w\") pod \"e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.300718 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.300686 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:02.443078 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.443035 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6"] Jul 20 05:39:02.445109 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:39:02.445049 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76c39f4_4868_4eb5_8e56_f2f7c3497321.slice/crio-f89d1eb7b70a597d013e0f2a21ee11bc1b0b1b6bdc1cc970a097c544d54fe372 WatchSource:0}: Error finding container f89d1eb7b70a597d013e0f2a21ee11bc1b0b1b6bdc1cc970a097c544d54fe372: Status 404 returned error can't find the container with id f89d1eb7b70a597d013e0f2a21ee11bc1b0b1b6bdc1cc970a097c544d54fe372 Jul 20 05:39:02.447396 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.447355 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:39:02.447474 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.447441 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:39:02.447516 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.447482 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:39:02.866691 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.866649 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" event={"ID":"a76c39f4-4868-4eb5-8e56-f2f7c3497321","Type":"ContainerStarted","Data":"f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1"} Jul 20 05:39:02.866896 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.866760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" event={"ID":"a76c39f4-4868-4eb5-8e56-f2f7c3497321","Type":"ContainerStarted","Data":"f89d1eb7b70a597d013e0f2a21ee11bc1b0b1b6bdc1cc970a097c544d54fe372"} Jul 20 05:39:02.888216 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:02.888141 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" podStartSLOduration=1.888122837 podStartE2EDuration="1.888122837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:39:02.885616148 +0000 UTC m=+2372.155099835" watchObservedRunningTime="2026-07-20 05:39:02.888122837 +0000 UTC m=+2372.157606522" Jul 20 05:39:03.023132 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:03.023098 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:39:03.301109 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:03.301052 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:03.305851 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:03.305828 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:03.875162 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:03.875128 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:03.880575 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:03.880538 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:10.835182 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:10.835154 2579 patch_prober.go:28] interesting pod/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 20 05:39:10.835511 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:10.835222 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 20 05:39:10.912232 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:10.912198 2579 generic.go:356] "Generic (PLEG): container finished" podID="d26154e2-5728-43ba-b19f-791db2f435ea" containerID="c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532" exitCode=0 Jul 20 05:39:10.912424 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:10.912274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" event={"ID":"d26154e2-5728-43ba-b19f-791db2f435ea","Type":"ContainerDied","Data":"c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532"} Jul 20 05:39:10.912762 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:10.912709 2579 scope.go:117] "RemoveContainer" containerID="c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532" Jul 20 05:39:10.913736 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:10.913706 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:39:10.913826 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:10.913793 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:39:10.913887 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:10.913842 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:39:11.921402 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:11.921371 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" event={"ID":"d26154e2-5728-43ba-b19f-791db2f435ea","Type":"ContainerStarted","Data":"e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755"} Jul 20 05:39:12.138693 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:12.138655 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:39:12.138868 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:12.138756 2579 patch_prober.go:28] interesting pod/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 20 05:39:12.138868 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:12.138799 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 20 05:39:13.138723 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:13.138692 2579 patch_prober.go:28] interesting pod/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 20 05:39:13.139192 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:13.139166 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 20 05:39:14.138790 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:14.138742 2579 patch_prober.go:28] interesting pod/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 20 05:39:14.139202 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:14.138819 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 20 05:39:15.138738 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:15.138706 2579 patch_prober.go:28] interesting pod/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 20 05:39:15.138897 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:15.138767 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 20 05:39:16.143797 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:16.143759 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:39:16.144228 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:16.144152 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:39:16.144958 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:16.144934 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:39:17.952005 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:17.951980 2579 generic.go:356] "Generic (PLEG): container finished" podID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerID="f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1" exitCode=0 Jul 20 05:39:17.952414 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:17.952069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" event={"ID":"a76c39f4-4868-4eb5-8e56-f2f7c3497321","Type":"ContainerDied","Data":"f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1"} Jul 20 05:39:17.952504 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:17.952489 2579 scope.go:117] "RemoveContainer" containerID="f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1" Jul 20 05:39:17.953425 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:17.953396 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:39:17.953516 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:17.953476 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:39:17.953578 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:17.953527 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:39:18.875683 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:18.875638 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:18.957950 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:18.957920 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" event={"ID":"a76c39f4-4868-4eb5-8e56-f2f7c3497321","Type":"ContainerStarted","Data":"25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356"} Jul 20 05:39:19.224658 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:19.224570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:39:19.301442 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:19.301407 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:19.302612 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:19.302583 2579 patch_prober.go:28] interesting pod/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 20 05:39:19.302736 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:19.302632 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 20 05:39:20.301582 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:20.301520 2579 patch_prober.go:28] interesting pod/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 20 05:39:20.301969 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:20.301600 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 20 05:39:21.302116 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:21.302051 2579 patch_prober.go:28] interesting pod/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 20 05:39:21.302603 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:21.302142 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 20 05:39:22.301355 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:22.301313 2579 patch_prober.go:28] interesting pod/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 20 05:39:22.301544 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:22.301381 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 20 05:39:23.305538 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:23.305503 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:23.305924 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:23.305816 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:23.306557 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:23.306542 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:39:30.936616 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:30.936577 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:39:40.728151 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.728118 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj"] Jul 20 05:39:40.731400 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.731380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.733859 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.733825 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-598bdb\"/\"test-model-598bdb-kserve-self-signed-certs\"" Jul 20 05:39:40.735370 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.735348 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-598bdb\"/\"kube-root-ca.crt\"" Jul 20 05:39:40.735493 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.735446 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-598bdb\"/\"default-dockercfg-5gjsh\"" Jul 20 05:39:40.735559 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.735511 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-598bdb\"/\"openshift-service-ca.crt\"" Jul 20 05:39:40.741997 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.741974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj"] Jul 20 05:39:40.864750 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.864714 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78nqw\" (UniqueName: \"kubernetes.io/projected/1bf43b47-538d-48f4-8cb1-480073b9389e-kube-api-access-78nqw\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.864918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.864810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-model-cache\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.864918 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.864885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-home\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.865022 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.864955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf43b47-538d-48f4-8cb1-480073b9389e-tls-certs\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.865022 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.864980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-dshm\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.865022 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.865008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-tmp-dir\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.965779 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.965741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78nqw\" (UniqueName: \"kubernetes.io/projected/1bf43b47-538d-48f4-8cb1-480073b9389e-kube-api-access-78nqw\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.965980 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.965805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-model-cache\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.965980 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.965851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-home\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.965980 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.965896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf43b47-538d-48f4-8cb1-480073b9389e-tls-certs\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.965980 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.965920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-dshm\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.965980 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.965950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-tmp-dir\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.966428 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.966403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-home\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.966428 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.966422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-tmp-dir\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.966689 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.966668 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-model-cache\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.968950 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.968914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-dshm\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.968950 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.968932 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf43b47-538d-48f4-8cb1-480073b9389e-tls-certs\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:40.973394 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:40.973373 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78nqw\" (UniqueName: \"kubernetes.io/projected/1bf43b47-538d-48f4-8cb1-480073b9389e-kube-api-access-78nqw\") pod \"test-model-598bdb-kserve-9cb7c78d5-5kgjj\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:41.040910 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:41.040879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:41.173137 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:41.173107 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj"] Jul 20 05:39:41.174365 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:39:41.174341 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bf43b47_538d_48f4_8cb1_480073b9389e.slice/crio-8827d8bcc5206a8911c0bc26e29e91a39babf01e27dd1a5cb63d404aebfb55f0 WatchSource:0}: Error finding container 8827d8bcc5206a8911c0bc26e29e91a39babf01e27dd1a5cb63d404aebfb55f0: Status 404 returned error can't find the container with id 8827d8bcc5206a8911c0bc26e29e91a39babf01e27dd1a5cb63d404aebfb55f0 Jul 20 05:39:42.070181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:42.070143 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" event={"ID":"1bf43b47-538d-48f4-8cb1-480073b9389e","Type":"ContainerStarted","Data":"76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632"} Jul 20 05:39:42.070181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:42.070187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" event={"ID":"1bf43b47-538d-48f4-8cb1-480073b9389e","Type":"ContainerStarted","Data":"8827d8bcc5206a8911c0bc26e29e91a39babf01e27dd1a5cb63d404aebfb55f0"} Jul 20 05:39:42.087014 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:42.086965 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" podStartSLOduration=2.086951016 podStartE2EDuration="2.086951016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:39:42.084960929 +0000 UTC m=+2411.354444614" watchObservedRunningTime="2026-07-20 05:39:42.086951016 +0000 UTC m=+2411.356434700" Jul 20 05:39:46.327475 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:46.327442 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:39:51.041254 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:51.041210 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:51.041659 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:51.041440 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:51.053862 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:51.053831 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:39:51.124356 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:39:51.124328 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:40:24.736210 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:24.736173 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj"] Jul 20 05:40:28.414820 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:28.414783 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:40:39.756224 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:39.756161 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" podUID="1bf43b47-538d-48f4-8cb1-480073b9389e" containerName="main" containerID="cri-o://76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632" gracePeriod=45 Jul 20 05:40:40.011079 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.011009 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:40:40.183605 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.183572 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf43b47-538d-48f4-8cb1-480073b9389e-tls-certs\") pod \"1bf43b47-538d-48f4-8cb1-480073b9389e\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " Jul 20 05:40:40.183781 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.183611 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-model-cache\") pod \"1bf43b47-538d-48f4-8cb1-480073b9389e\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " Jul 20 05:40:40.183781 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.183680 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-tmp-dir\") pod \"1bf43b47-538d-48f4-8cb1-480073b9389e\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " Jul 20 05:40:40.183781 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.183719 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-home\") pod \"1bf43b47-538d-48f4-8cb1-480073b9389e\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " Jul 20 05:40:40.183781 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.183741 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-dshm\") pod \"1bf43b47-538d-48f4-8cb1-480073b9389e\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " Jul 20 05:40:40.183781 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.183769 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78nqw\" (UniqueName: \"kubernetes.io/projected/1bf43b47-538d-48f4-8cb1-480073b9389e-kube-api-access-78nqw\") pod \"1bf43b47-538d-48f4-8cb1-480073b9389e\" (UID: \"1bf43b47-538d-48f4-8cb1-480073b9389e\") " Jul 20 05:40:40.184079 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.183932 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bf43b47-538d-48f4-8cb1-480073b9389e" (UID: "1bf43b47-538d-48f4-8cb1-480073b9389e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:40:40.184079 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.183950 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bf43b47-538d-48f4-8cb1-480073b9389e" (UID: "1bf43b47-538d-48f4-8cb1-480073b9389e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:40:40.184079 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.184004 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-home" (OuterVolumeSpecName: "home") pod "1bf43b47-538d-48f4-8cb1-480073b9389e" (UID: "1bf43b47-538d-48f4-8cb1-480073b9389e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:40:40.184232 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.184133 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-model-cache\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:40.184232 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.184154 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-tmp-dir\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:40.184232 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.184170 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-home\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:40.185870 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.185848 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-dshm" (OuterVolumeSpecName: "dshm") pod "1bf43b47-538d-48f4-8cb1-480073b9389e" (UID: "1bf43b47-538d-48f4-8cb1-480073b9389e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:40:40.185942 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.185879 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf43b47-538d-48f4-8cb1-480073b9389e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bf43b47-538d-48f4-8cb1-480073b9389e" (UID: "1bf43b47-538d-48f4-8cb1-480073b9389e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 20 05:40:40.185942 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.185924 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf43b47-538d-48f4-8cb1-480073b9389e-kube-api-access-78nqw" (OuterVolumeSpecName: "kube-api-access-78nqw") pod "1bf43b47-538d-48f4-8cb1-480073b9389e" (UID: "1bf43b47-538d-48f4-8cb1-480073b9389e"). InnerVolumeSpecName "kube-api-access-78nqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:40:40.284948 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.284916 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bf43b47-538d-48f4-8cb1-480073b9389e-tls-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:40.284948 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.284944 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bf43b47-538d-48f4-8cb1-480073b9389e-dshm\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:40.284948 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.284953 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78nqw\" (UniqueName: \"kubernetes.io/projected/1bf43b47-538d-48f4-8cb1-480073b9389e-kube-api-access-78nqw\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:40.339147 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.339112 2579 generic.go:356] "Generic (PLEG): container finished" podID="1bf43b47-538d-48f4-8cb1-480073b9389e" containerID="76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632" exitCode=0 Jul 20 05:40:40.339288 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.339160 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" Jul 20 05:40:40.339288 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.339173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" event={"ID":"1bf43b47-538d-48f4-8cb1-480073b9389e","Type":"ContainerDied","Data":"76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632"} Jul 20 05:40:40.339288 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.339204 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj" event={"ID":"1bf43b47-538d-48f4-8cb1-480073b9389e","Type":"ContainerDied","Data":"8827d8bcc5206a8911c0bc26e29e91a39babf01e27dd1a5cb63d404aebfb55f0"} Jul 20 05:40:40.339288 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.339221 2579 scope.go:117] "RemoveContainer" containerID="76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632" Jul 20 05:40:40.347783 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.347761 2579 scope.go:117] "RemoveContainer" containerID="76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632" Jul 20 05:40:40.348027 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:40:40.348007 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632\": container with ID starting with 76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632 not found: ID does not exist" containerID="76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632" Jul 20 05:40:40.348091 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.348035 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632"} err="failed to get container status \"76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632\": rpc error: code = NotFound desc = could not find container \"76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632\": container with ID starting with 76b77e902e4ca0f15edad10729e1f950e09ecf38287856f980288446f8fa2632 not found: ID does not exist" Jul 20 05:40:40.359102 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.359077 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj"] Jul 20 05:40:40.362731 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.362708 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-598bdb/test-model-598bdb-kserve-9cb7c78d5-5kgjj"] Jul 20 05:40:40.530115 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:40.530077 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:40:41.155933 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:41.155900 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf43b47-538d-48f4-8cb1-480073b9389e" path="/var/lib/kubelet/pods/1bf43b47-538d-48f4-8cb1-480073b9389e/volumes" Jul 20 05:40:46.617773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:46.617738 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7"] Jul 20 05:40:46.618173 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:46.618045 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" containerID="cri-o://e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755" gracePeriod=30 Jul 20 05:40:51.877555 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.877486 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:40:51.893670 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.893648 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-istio-token\") pod \"d26154e2-5728-43ba-b19f-791db2f435ea\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " Jul 20 05:40:51.893803 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.893684 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-certs\") pod \"d26154e2-5728-43ba-b19f-791db2f435ea\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " Jul 20 05:40:51.893803 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.893706 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-data\") pod \"d26154e2-5728-43ba-b19f-791db2f435ea\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " Jul 20 05:40:51.893803 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.893755 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26154e2-5728-43ba-b19f-791db2f435ea-istiod-ca-cert\") pod \"d26154e2-5728-43ba-b19f-791db2f435ea\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " Jul 20 05:40:51.893803 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.893783 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-755ln\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-kube-api-access-755ln\") pod \"d26154e2-5728-43ba-b19f-791db2f435ea\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " Jul 20 05:40:51.894012 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.893864 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-credential-socket\") pod \"d26154e2-5728-43ba-b19f-791db2f435ea\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " Jul 20 05:40:51.894012 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.893904 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26154e2-5728-43ba-b19f-791db2f435ea-istio-podinfo\") pod \"d26154e2-5728-43ba-b19f-791db2f435ea\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " Jul 20 05:40:51.894012 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.893951 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d26154e2-5728-43ba-b19f-791db2f435ea" (UID: "d26154e2-5728-43ba-b19f-791db2f435ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:40:51.894012 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.893959 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-socket\") pod \"d26154e2-5728-43ba-b19f-791db2f435ea\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " Jul 20 05:40:51.894266 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.894078 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-envoy\") pod \"d26154e2-5728-43ba-b19f-791db2f435ea\" (UID: \"d26154e2-5728-43ba-b19f-791db2f435ea\") " Jul 20 05:40:51.894266 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.894174 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26154e2-5728-43ba-b19f-791db2f435ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d26154e2-5728-43ba-b19f-791db2f435ea" (UID: "d26154e2-5728-43ba-b19f-791db2f435ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:40:51.894266 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.894182 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d26154e2-5728-43ba-b19f-791db2f435ea" (UID: "d26154e2-5728-43ba-b19f-791db2f435ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:40:51.894425 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.894380 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d26154e2-5728-43ba-b19f-791db2f435ea" (UID: "d26154e2-5728-43ba-b19f-791db2f435ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:40:51.894531 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.894511 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:51.894595 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.894535 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:51.894595 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.894550 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:51.894595 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.894562 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26154e2-5728-43ba-b19f-791db2f435ea-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:51.894820 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.894794 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "d26154e2-5728-43ba-b19f-791db2f435ea" (UID: "d26154e2-5728-43ba-b19f-791db2f435ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:40:51.896256 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.896229 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d26154e2-5728-43ba-b19f-791db2f435ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d26154e2-5728-43ba-b19f-791db2f435ea" (UID: "d26154e2-5728-43ba-b19f-791db2f435ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:40:51.896381 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.896291 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d26154e2-5728-43ba-b19f-791db2f435ea" (UID: "d26154e2-5728-43ba-b19f-791db2f435ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:40:51.896452 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.896401 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "d26154e2-5728-43ba-b19f-791db2f435ea" (UID: "d26154e2-5728-43ba-b19f-791db2f435ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:40:51.896689 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.896670 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-kube-api-access-755ln" (OuterVolumeSpecName: "kube-api-access-755ln") pod "d26154e2-5728-43ba-b19f-791db2f435ea" (UID: "d26154e2-5728-43ba-b19f-791db2f435ea"). InnerVolumeSpecName "kube-api-access-755ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:40:51.995874 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.995841 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26154e2-5728-43ba-b19f-791db2f435ea-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:51.995874 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.995868 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:51.995874 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.995879 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:51.996128 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.995887 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26154e2-5728-43ba-b19f-791db2f435ea-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:51.996128 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:51.995897 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-755ln\" (UniqueName: \"kubernetes.io/projected/d26154e2-5728-43ba-b19f-791db2f435ea-kube-api-access-755ln\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:40:52.395691 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.395656 2579 generic.go:356] "Generic (PLEG): container finished" podID="d26154e2-5728-43ba-b19f-791db2f435ea" containerID="e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755" exitCode=0 Jul 20 05:40:52.395876 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.395708 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" Jul 20 05:40:52.395876 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.395734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" event={"ID":"d26154e2-5728-43ba-b19f-791db2f435ea","Type":"ContainerDied","Data":"e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755"} Jul 20 05:40:52.395876 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.395790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7" event={"ID":"d26154e2-5728-43ba-b19f-791db2f435ea","Type":"ContainerDied","Data":"b750c77c43c91c441d0ce9c6e90cb4e94922a5b21234d46206fb8169aa82b998"} Jul 20 05:40:52.395876 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.395811 2579 scope.go:117] "RemoveContainer" containerID="e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755" Jul 20 05:40:52.405247 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.405228 2579 scope.go:117] "RemoveContainer" containerID="c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532" Jul 20 05:40:52.412597 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.412580 2579 scope.go:117] "RemoveContainer" containerID="e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755" Jul 20 05:40:52.412825 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:40:52.412809 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755\": container with ID starting with e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755 not found: ID does not exist" containerID="e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755" Jul 20 05:40:52.412874 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.412834 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755"} err="failed to get container status \"e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755\": rpc error: code = NotFound desc = could not find container \"e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755\": container with ID starting with e3504c4cdfb7d619deb5ba47993dc556a306cc395dc8799663fd2c30e0b84755 not found: ID does not exist" Jul 20 05:40:52.412874 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.412855 2579 scope.go:117] "RemoveContainer" containerID="c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532" Jul 20 05:40:52.413048 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:40:52.413034 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532\": container with ID starting with c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532 not found: ID does not exist" containerID="c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532" Jul 20 05:40:52.413119 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.413051 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532"} err="failed to get container status \"c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532\": rpc error: code = NotFound desc = could not find container \"c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532\": container with ID starting with c41fe40e626f7bd735334ba16cc8b595a284e11b271c12fc799f6bef3d3ab532 not found: ID does not exist" Jul 20 05:40:52.417673 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.417652 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7"] Jul 20 05:40:52.423129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:52.423108 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-192e48-openshift-default-fb487c475-ztkk7"] Jul 20 05:40:53.155382 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:40:53.155351 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" path="/var/lib/kubelet/pods/d26154e2-5728-43ba-b19f-791db2f435ea/volumes" Jul 20 05:41:00.525884 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:00.525853 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:41:08.388811 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:08.388769 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6"] Jul 20 05:41:08.389247 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:08.389084 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" containerID="cri-o://25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356" gracePeriod=30 Jul 20 05:41:09.335281 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.335247 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv"] Jul 20 05:41:09.335590 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.335578 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" Jul 20 05:41:09.335634 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.335591 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" Jul 20 05:41:09.335634 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.335599 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf43b47-538d-48f4-8cb1-480073b9389e" containerName="main" Jul 20 05:41:09.335634 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.335605 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf43b47-538d-48f4-8cb1-480073b9389e" containerName="main" Jul 20 05:41:09.335634 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.335617 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" Jul 20 05:41:09.335634 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.335623 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" Jul 20 05:41:09.335785 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.335675 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" Jul 20 05:41:09.335785 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.335684 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf43b47-538d-48f4-8cb1-480073b9389e" containerName="main" Jul 20 05:41:09.338610 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.338594 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.343757 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.343734 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ce4188-openshift-default-dockercfg-st6zn\"" Jul 20 05:41:09.370677 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.370648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv"] Jul 20 05:41:09.441651 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.441615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-envoy\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.441651 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.441654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istiod-ca-cert\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.442129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.441684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-podinfo\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.442129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.441751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-credential-socket\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.442129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.441791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-certs\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.442129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.441814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm4pl\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-kube-api-access-bm4pl\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.442129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.441841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-data\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.442129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.441856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-token\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.442129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.441890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-socket\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543044 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-data\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543044 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-token\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-socket\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-envoy\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istiod-ca-cert\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-podinfo\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-credential-socket\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-certs\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm4pl\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-kube-api-access-bm4pl\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543663 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-socket\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543663 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543655 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-credential-socket\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543758 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-certs\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543796 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-data\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.543999 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.543968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istiod-ca-cert\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.545624 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.545603 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-envoy\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.545820 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.545803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-podinfo\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.552155 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.552124 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-token\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.552290 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.552165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm4pl\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-kube-api-access-bm4pl\") pod \"e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.648856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.648766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:09.789321 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.789292 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv"] Jul 20 05:41:09.792916 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:41:09.792883 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c7167f_5210_4da7_b9ca_fd90a92b372f.slice/crio-0f2b060a00656db48882eec4194e26023545339006e13c4942c23fd70f4c3dd4 WatchSource:0}: Error finding container 0f2b060a00656db48882eec4194e26023545339006e13c4942c23fd70f4c3dd4: Status 404 returned error can't find the container with id 0f2b060a00656db48882eec4194e26023545339006e13c4942c23fd70f4c3dd4 Jul 20 05:41:09.795202 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.795168 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:09.795283 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.795240 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:09.795283 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:09.795270 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:10.480769 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:10.480735 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" event={"ID":"d2c7167f-5210-4da7-b9ca-fd90a92b372f","Type":"ContainerStarted","Data":"e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26"} Jul 20 05:41:10.481188 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:10.480796 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" event={"ID":"d2c7167f-5210-4da7-b9ca-fd90a92b372f","Type":"ContainerStarted","Data":"0f2b060a00656db48882eec4194e26023545339006e13c4942c23fd70f4c3dd4"} Jul 20 05:41:10.500818 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:10.500766 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" podStartSLOduration=1.500749181 podStartE2EDuration="1.500749181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:41:10.49909633 +0000 UTC m=+2499.768580013" watchObservedRunningTime="2026-07-20 05:41:10.500749181 +0000 UTC m=+2499.770232866" Jul 20 05:41:10.649620 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:10.649581 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:10.654286 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:10.654256 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:11.485320 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:11.485287 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:11.486218 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:11.486195 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:13.639465 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.639440 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:41:13.784300 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784271 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istiod-ca-cert\") pod \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " Jul 20 05:41:13.784475 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784324 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-envoy\") pod \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " Jul 20 05:41:13.784475 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784347 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-token\") pod \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " Jul 20 05:41:13.784475 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784385 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-certs\") pod \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " Jul 20 05:41:13.784475 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784410 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqt8w\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-kube-api-access-vqt8w\") pod \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " Jul 20 05:41:13.784475 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784453 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-data\") pod \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " Jul 20 05:41:13.784785 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784488 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-socket\") pod \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " Jul 20 05:41:13.784785 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784511 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-podinfo\") pod \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " Jul 20 05:41:13.784785 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784543 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-credential-socket\") pod \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\" (UID: \"a76c39f4-4868-4eb5-8e56-f2f7c3497321\") " Jul 20 05:41:13.784948 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784805 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a76c39f4-4868-4eb5-8e56-f2f7c3497321" (UID: "a76c39f4-4868-4eb5-8e56-f2f7c3497321"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:41:13.784948 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784811 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a76c39f4-4868-4eb5-8e56-f2f7c3497321" (UID: "a76c39f4-4868-4eb5-8e56-f2f7c3497321"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:41:13.785083 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784954 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a76c39f4-4868-4eb5-8e56-f2f7c3497321" (UID: "a76c39f4-4868-4eb5-8e56-f2f7c3497321"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:41:13.785083 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.784968 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a76c39f4-4868-4eb5-8e56-f2f7c3497321" (UID: "a76c39f4-4868-4eb5-8e56-f2f7c3497321"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:41:13.785400 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.785377 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-data" (OuterVolumeSpecName: "istio-data") pod "a76c39f4-4868-4eb5-8e56-f2f7c3497321" (UID: "a76c39f4-4868-4eb5-8e56-f2f7c3497321"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:41:13.786782 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.786759 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-kube-api-access-vqt8w" (OuterVolumeSpecName: "kube-api-access-vqt8w") pod "a76c39f4-4868-4eb5-8e56-f2f7c3497321" (UID: "a76c39f4-4868-4eb5-8e56-f2f7c3497321"). InnerVolumeSpecName "kube-api-access-vqt8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:41:13.786884 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.786831 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a76c39f4-4868-4eb5-8e56-f2f7c3497321" (UID: "a76c39f4-4868-4eb5-8e56-f2f7c3497321"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:41:13.787093 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.787038 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a76c39f4-4868-4eb5-8e56-f2f7c3497321" (UID: "a76c39f4-4868-4eb5-8e56-f2f7c3497321"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:41:13.787164 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.787082 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-token" (OuterVolumeSpecName: "istio-token") pod "a76c39f4-4868-4eb5-8e56-f2f7c3497321" (UID: "a76c39f4-4868-4eb5-8e56-f2f7c3497321"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:41:13.886221 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.886182 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:41:13.886221 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.886216 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:41:13.886221 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.886225 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:41:13.886221 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.886233 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:41:13.886483 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.886242 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqt8w\" (UniqueName: \"kubernetes.io/projected/a76c39f4-4868-4eb5-8e56-f2f7c3497321-kube-api-access-vqt8w\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:41:13.886483 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.886251 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:41:13.886483 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.886260 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:41:13.886483 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.886268 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76c39f4-4868-4eb5-8e56-f2f7c3497321-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:41:13.886483 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:13.886276 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76c39f4-4868-4eb5-8e56-f2f7c3497321-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:41:14.499887 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.499854 2579 generic.go:356] "Generic (PLEG): container finished" podID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerID="25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356" exitCode=0 Jul 20 05:41:14.500078 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.499902 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" Jul 20 05:41:14.500078 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.499950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" event={"ID":"a76c39f4-4868-4eb5-8e56-f2f7c3497321","Type":"ContainerDied","Data":"25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356"} Jul 20 05:41:14.500078 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.500005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6" event={"ID":"a76c39f4-4868-4eb5-8e56-f2f7c3497321","Type":"ContainerDied","Data":"f89d1eb7b70a597d013e0f2a21ee11bc1b0b1b6bdc1cc970a097c544d54fe372"} Jul 20 05:41:14.500078 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.500027 2579 scope.go:117] "RemoveContainer" containerID="25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356" Jul 20 05:41:14.508944 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.508926 2579 scope.go:117] "RemoveContainer" containerID="f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1" Jul 20 05:41:14.516300 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.516283 2579 scope.go:117] "RemoveContainer" containerID="25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356" Jul 20 05:41:14.516555 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:41:14.516524 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356\": container with ID starting with 25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356 not found: ID does not exist" containerID="25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356" Jul 20 05:41:14.516635 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.516558 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356"} err="failed to get container status \"25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356\": rpc error: code = NotFound desc = could not find container \"25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356\": container with ID starting with 25ebbf3e194bbc37ed1033bebfb22195d2b5abc6480e74897c6f079e4fa45356 not found: ID does not exist" Jul 20 05:41:14.516635 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.516577 2579 scope.go:117] "RemoveContainer" containerID="f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1" Jul 20 05:41:14.516862 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:41:14.516839 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1\": container with ID starting with f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1 not found: ID does not exist" containerID="f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1" Jul 20 05:41:14.516906 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.516871 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1"} err="failed to get container status \"f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1\": rpc error: code = NotFound desc = could not find container \"f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1\": container with ID starting with f6bd727116f737ee56e7bc82440488982a5ce7bef1ff5da50f4eca4e117030c1 not found: ID does not exist" Jul 20 05:41:14.538073 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.538039 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6"] Jul 20 05:41:14.541000 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:14.540979 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-598bdb-openshift-default-7c4f9cf867-grjt6"] Jul 20 05:41:15.155113 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:15.155045 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" path="/var/lib/kubelet/pods/a76c39f4-4868-4eb5-8e56-f2f7c3497321/volumes" Jul 20 05:41:17.737815 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.737782 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n"] Jul 20 05:41:17.738173 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.738120 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" Jul 20 05:41:17.738173 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.738132 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" Jul 20 05:41:17.738173 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.738142 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" Jul 20 05:41:17.738173 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.738148 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" Jul 20 05:41:17.738322 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.738207 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26154e2-5728-43ba-b19f-791db2f435ea" containerName="istio-proxy" Jul 20 05:41:17.738322 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.738219 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" Jul 20 05:41:17.738322 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.738225 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76c39f4-4868-4eb5-8e56-f2f7c3497321" containerName="istio-proxy" Jul 20 05:41:17.742372 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.742354 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.745634 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.745610 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d0511a-openshift-default-dockercfg-4nclh\"" Jul 20 05:41:17.762685 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.762654 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n"] Jul 20 05:41:17.821873 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.821834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4c4\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-kube-api-access-rt4c4\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.822069 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.821886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-token\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.822069 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.821915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f31482-65f6-4cc8-925e-aaacd7045cf5-istiod-ca-cert\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.822069 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.821932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-socket\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.822069 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.821947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-credential-socket\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.822069 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.821961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-certs\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.822069 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.822006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-envoy\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.822308 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.822081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-podinfo\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.822308 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.822111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-data\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923163 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-podinfo\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923163 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-data\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923411 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4c4\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-kube-api-access-rt4c4\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923411 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-token\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923411 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f31482-65f6-4cc8-925e-aaacd7045cf5-istiod-ca-cert\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923411 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-socket\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923411 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-credential-socket\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923411 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-certs\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923411 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-envoy\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923805 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-credential-socket\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923874 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-certs\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.923923 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-socket\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.924008 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.923989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-data\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.924103 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.924086 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f31482-65f6-4cc8-925e-aaacd7045cf5-istiod-ca-cert\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.925716 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.925686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-envoy\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.925716 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.925710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-podinfo\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.942009 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.941987 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4c4\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-kube-api-access-rt4c4\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:17.947333 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:17.947316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-token\") pod \"e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:18.058276 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:18.058240 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:18.219027 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:18.219002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n"] Jul 20 05:41:18.221376 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:41:18.221348 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f31482_65f6_4cc8_925e_aaacd7045cf5.slice/crio-c51c744aa807c2e2293c8a339943fc04db8b8ef134e14023689e773b86ecdf4e WatchSource:0}: Error finding container c51c744aa807c2e2293c8a339943fc04db8b8ef134e14023689e773b86ecdf4e: Status 404 returned error can't find the container with id c51c744aa807c2e2293c8a339943fc04db8b8ef134e14023689e773b86ecdf4e Jul 20 05:41:18.223371 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:18.223337 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:18.223452 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:18.223407 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:18.223452 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:18.223437 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:18.520968 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:18.520936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" event={"ID":"02f31482-65f6-4cc8-925e-aaacd7045cf5","Type":"ContainerStarted","Data":"5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d"} Jul 20 05:41:18.521171 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:18.520989 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" event={"ID":"02f31482-65f6-4cc8-925e-aaacd7045cf5","Type":"ContainerStarted","Data":"c51c744aa807c2e2293c8a339943fc04db8b8ef134e14023689e773b86ecdf4e"} Jul 20 05:41:18.559729 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:18.559677 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" podStartSLOduration=1.5596607919999999 podStartE2EDuration="1.559660792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:41:18.558661885 +0000 UTC m=+2507.828145571" watchObservedRunningTime="2026-07-20 05:41:18.559660792 +0000 UTC m=+2507.829144479" Jul 20 05:41:19.058897 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:19.058858 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:19.064010 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:19.063986 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:19.526127 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:19.526103 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:19.527180 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:19.527160 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:20.121905 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:20.121871 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:41:27.527238 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:27.527192 2579 patch_prober.go:28] interesting pod/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 20 05:41:27.527693 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:27.527266 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 20 05:41:28.570783 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:28.570749 2579 generic.go:356] "Generic (PLEG): container finished" podID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerID="e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26" exitCode=0 Jul 20 05:41:28.571343 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:28.570832 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" event={"ID":"d2c7167f-5210-4da7-b9ca-fd90a92b372f","Type":"ContainerDied","Data":"e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26"} Jul 20 05:41:28.571343 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:28.571275 2579 scope.go:117] "RemoveContainer" containerID="e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26" Jul 20 05:41:28.572279 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:28.572248 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:28.572372 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:28.572318 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:28.572372 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:28.572353 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:29.594618 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:29.594566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" event={"ID":"d2c7167f-5210-4da7-b9ca-fd90a92b372f","Type":"ContainerStarted","Data":"a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda"} Jul 20 05:41:29.649931 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:29.649892 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:29.650131 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:29.650095 2579 patch_prober.go:28] interesting pod/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 20 05:41:29.650205 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:29.650140 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 20 05:41:30.649586 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:30.649548 2579 patch_prober.go:28] interesting pod/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 20 05:41:30.650020 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:30.649626 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 20 05:41:31.649609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:31.649510 2579 patch_prober.go:28] interesting pod/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 20 05:41:31.649609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:31.649573 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 20 05:41:32.649727 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:32.649683 2579 patch_prober.go:28] interesting pod/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 20 05:41:32.650267 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:32.649766 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 20 05:41:33.652417 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:33.652389 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:33.652808 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:33.652729 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:33.653353 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:33.653334 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:41:36.917270 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:36.917233 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:41:37.643825 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:37.643790 2579 generic.go:356] "Generic (PLEG): container finished" podID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerID="5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d" exitCode=0 Jul 20 05:41:37.644018 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:37.643841 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" event={"ID":"02f31482-65f6-4cc8-925e-aaacd7045cf5","Type":"ContainerDied","Data":"5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d"} Jul 20 05:41:37.644280 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:37.644265 2579 scope.go:117] "RemoveContainer" containerID="5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d" Jul 20 05:41:37.645320 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:37.645284 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:37.645453 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:37.645381 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:37.645453 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:37.645431 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:41:38.651626 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:38.651597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" event={"ID":"02f31482-65f6-4cc8-925e-aaacd7045cf5","Type":"ContainerStarted","Data":"2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc"} Jul 20 05:41:39.058647 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:39.058606 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:39.058832 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:39.058813 2579 patch_prober.go:28] interesting pod/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 20 05:41:39.058896 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:39.058861 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 20 05:41:40.058847 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:40.058809 2579 patch_prober.go:28] interesting pod/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 20 05:41:40.059282 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:40.058863 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 20 05:41:41.059523 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:41.059488 2579 patch_prober.go:28] interesting pod/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 20 05:41:41.059916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:41.059560 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 20 05:41:42.058591 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:42.058554 2579 patch_prober.go:28] interesting pod/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 20 05:41:42.058769 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:42.058621 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 20 05:41:43.062797 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:43.062766 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:43.063213 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:43.063085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:43.063733 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:43.063713 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:41:52.524267 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:52.524229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:41:59.395183 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:59.395148 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv"] Jul 20 05:41:59.396588 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:41:59.395935 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" containerID="cri-o://a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda" gracePeriod=30 Jul 20 05:42:03.655493 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:03.655460 2579 patch_prober.go:28] interesting pod/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 05:42:03.655974 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:03.655524 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 05:42:04.648200 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.648176 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:42:04.755201 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755115 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-podinfo\") pod \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " Jul 20 05:42:04.755201 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755160 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-certs\") pod \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " Jul 20 05:42:04.755201 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755181 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-data\") pod \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " Jul 20 05:42:04.755773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755224 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-token\") pod \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " Jul 20 05:42:04.755773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755243 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm4pl\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-kube-api-access-bm4pl\") pod \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " Jul 20 05:42:04.755773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755274 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-envoy\") pod \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " Jul 20 05:42:04.755773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755306 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-socket\") pod \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " Jul 20 05:42:04.755773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755333 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-credential-socket\") pod \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " Jul 20 05:42:04.755773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istiod-ca-cert\") pod \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\" (UID: \"d2c7167f-5210-4da7-b9ca-fd90a92b372f\") " Jul 20 05:42:04.755773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755480 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2c7167f-5210-4da7-b9ca-fd90a92b372f" (UID: "d2c7167f-5210-4da7-b9ca-fd90a92b372f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:04.755773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755644 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2c7167f-5210-4da7-b9ca-fd90a92b372f" (UID: "d2c7167f-5210-4da7-b9ca-fd90a92b372f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:04.756219 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755791 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:04.756219 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755811 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:04.756219 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755866 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2c7167f-5210-4da7-b9ca-fd90a92b372f" (UID: "d2c7167f-5210-4da7-b9ca-fd90a92b372f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:42:04.756219 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.755885 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2c7167f-5210-4da7-b9ca-fd90a92b372f" (UID: "d2c7167f-5210-4da7-b9ca-fd90a92b372f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:04.756443 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.756328 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2c7167f-5210-4da7-b9ca-fd90a92b372f" (UID: "d2c7167f-5210-4da7-b9ca-fd90a92b372f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:04.757471 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.757448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2c7167f-5210-4da7-b9ca-fd90a92b372f" (UID: "d2c7167f-5210-4da7-b9ca-fd90a92b372f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:42:04.757579 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.757498 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2c7167f-5210-4da7-b9ca-fd90a92b372f" (UID: "d2c7167f-5210-4da7-b9ca-fd90a92b372f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:04.757648 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.757625 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2c7167f-5210-4da7-b9ca-fd90a92b372f" (UID: "d2c7167f-5210-4da7-b9ca-fd90a92b372f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:42:04.757688 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.757662 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-kube-api-access-bm4pl" (OuterVolumeSpecName: "kube-api-access-bm4pl") pod "d2c7167f-5210-4da7-b9ca-fd90a92b372f" (UID: "d2c7167f-5210-4da7-b9ca-fd90a92b372f"). InnerVolumeSpecName "kube-api-access-bm4pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:42:04.773151 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.773119 2579 generic.go:356] "Generic (PLEG): container finished" podID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerID="a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda" exitCode=0 Jul 20 05:42:04.773272 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.773168 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" Jul 20 05:42:04.773272 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.773206 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" event={"ID":"d2c7167f-5210-4da7-b9ca-fd90a92b372f","Type":"ContainerDied","Data":"a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda"} Jul 20 05:42:04.773398 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.773271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv" event={"ID":"d2c7167f-5210-4da7-b9ca-fd90a92b372f","Type":"ContainerDied","Data":"0f2b060a00656db48882eec4194e26023545339006e13c4942c23fd70f4c3dd4"} Jul 20 05:42:04.773398 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.773306 2579 scope.go:117] "RemoveContainer" containerID="a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda" Jul 20 05:42:04.782113 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.782097 2579 scope.go:117] "RemoveContainer" containerID="e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26" Jul 20 05:42:04.789564 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.789546 2579 scope.go:117] "RemoveContainer" containerID="a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda" Jul 20 05:42:04.789844 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:42:04.789826 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda\": container with ID starting with a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda not found: ID does not exist" containerID="a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda" Jul 20 05:42:04.789890 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.789852 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda"} err="failed to get container status \"a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda\": rpc error: code = NotFound desc = could not find container \"a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda\": container with ID starting with a5df61ac033b4c2cf0760db514e7329141d3c0315e162739a7eabd1626254cda not found: ID does not exist" Jul 20 05:42:04.789890 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.789868 2579 scope.go:117] "RemoveContainer" containerID="e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26" Jul 20 05:42:04.790135 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:42:04.790113 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26\": container with ID starting with e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26 not found: ID does not exist" containerID="e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26" Jul 20 05:42:04.790214 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.790145 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26"} err="failed to get container status \"e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26\": rpc error: code = NotFound desc = could not find container \"e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26\": container with ID starting with e5ea3c9d86be722a8b1a2d790353cda4779c49532aec2a8dc27fcb341f20fc26 not found: ID does not exist" Jul 20 05:42:04.795373 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.795351 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv"] Jul 20 05:42:04.799210 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.799190 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ce4188-openshift-default-ccf8bf589-lhvmv"] Jul 20 05:42:04.856895 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.856870 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:04.856895 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.856894 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm4pl\" (UniqueName: \"kubernetes.io/projected/d2c7167f-5210-4da7-b9ca-fd90a92b372f-kube-api-access-bm4pl\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:04.857029 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.856906 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:04.857029 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.856916 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:04.857029 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.856924 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:04.857029 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.856933 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:04.857029 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:04.856940 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c7167f-5210-4da7-b9ca-fd90a92b372f-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:05.155173 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:05.155142 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" path="/var/lib/kubelet/pods/d2c7167f-5210-4da7-b9ca-fd90a92b372f/volumes" Jul 20 05:42:20.025078 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:20.025034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:42:26.058701 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.058653 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n"] Jul 20 05:42:26.059170 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.058923 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" containerID="cri-o://2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc" gracePeriod=30 Jul 20 05:42:26.910866 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.910828 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2"] Jul 20 05:42:26.911262 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.911244 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" Jul 20 05:42:26.911362 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.911264 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" Jul 20 05:42:26.911419 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.911373 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" Jul 20 05:42:26.911419 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.911387 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" Jul 20 05:42:26.914475 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.914452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:26.916896 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.916874 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-851aef-openshift-default-dockercfg-mm857\"" Jul 20 05:42:26.925808 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.925787 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2"] Jul 20 05:42:26.936697 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.936670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-credential-socket\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:26.936804 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.936700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-data\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:26.936804 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.936736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-envoy\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:26.936891 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.936810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-token\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:26.936891 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.936837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr99l\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-kube-api-access-dr99l\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:26.936891 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.936863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-socket\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:26.936891 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.936880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istiod-ca-cert\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:26.937031 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.936917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-certs\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:26.937031 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:26.936944 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-podinfo\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038072 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-token\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038072 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr99l\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-kube-api-access-dr99l\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-socket\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istiod-ca-cert\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-certs\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-podinfo\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-credential-socket\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-data\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038352 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-envoy\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038708 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038543 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-socket\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038708 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-certs\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038708 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-data\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.038829 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.038707 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-credential-socket\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.039024 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.039005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istiod-ca-cert\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.040622 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.040602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-envoy\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.040788 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.040770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-podinfo\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.045970 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.045946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr99l\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-kube-api-access-dr99l\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.046077 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.045995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-token\") pod \"e2e-shared-a-851aef-openshift-default-69f7799847-2twb2\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.224533 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.224455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:27.360434 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.360409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2"] Jul 20 05:42:27.363153 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:42:27.363121 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de49bff_8ae4_45ca_b7fd_5c3557211c65.slice/crio-1f0483365b64f32138557b819bc5a057b31bb0049fb42efde4e75517622c82d0 WatchSource:0}: Error finding container 1f0483365b64f32138557b819bc5a057b31bb0049fb42efde4e75517622c82d0: Status 404 returned error can't find the container with id 1f0483365b64f32138557b819bc5a057b31bb0049fb42efde4e75517622c82d0 Jul 20 05:42:27.365987 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.365947 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:27.366115 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.366035 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:27.366115 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.366095 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:27.882481 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.882441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" event={"ID":"7de49bff-8ae4-45ca-b7fd-5c3557211c65","Type":"ContainerStarted","Data":"b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47"} Jul 20 05:42:27.882655 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.882508 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" event={"ID":"7de49bff-8ae4-45ca-b7fd-5c3557211c65","Type":"ContainerStarted","Data":"1f0483365b64f32138557b819bc5a057b31bb0049fb42efde4e75517622c82d0"} Jul 20 05:42:27.902085 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:27.902024 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" podStartSLOduration=1.9020087 podStartE2EDuration="1.9020087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:42:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:42:27.900390538 +0000 UTC m=+2577.169874224" watchObservedRunningTime="2026-07-20 05:42:27.9020087 +0000 UTC m=+2577.171492385" Jul 20 05:42:28.066297 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:28.066254 2579 patch_prober.go:28] interesting pod/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 20 05:42:28.066463 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:28.066322 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 20 05:42:28.225345 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:28.225250 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:28.230170 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:28.230147 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:28.887659 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:28.887621 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:28.888637 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:28.888616 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:31.312217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.312187 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:42:31.378521 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378440 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-envoy\") pod \"02f31482-65f6-4cc8-925e-aaacd7045cf5\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " Jul 20 05:42:31.378521 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378498 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-certs\") pod \"02f31482-65f6-4cc8-925e-aaacd7045cf5\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " Jul 20 05:42:31.378733 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378525 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-credential-socket\") pod \"02f31482-65f6-4cc8-925e-aaacd7045cf5\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " Jul 20 05:42:31.378733 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378543 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-podinfo\") pod \"02f31482-65f6-4cc8-925e-aaacd7045cf5\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " Jul 20 05:42:31.378733 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378563 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-token\") pod \"02f31482-65f6-4cc8-925e-aaacd7045cf5\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " Jul 20 05:42:31.378733 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378592 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-socket\") pod \"02f31482-65f6-4cc8-925e-aaacd7045cf5\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " Jul 20 05:42:31.378733 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378631 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-data\") pod \"02f31482-65f6-4cc8-925e-aaacd7045cf5\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " Jul 20 05:42:31.378733 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378685 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt4c4\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-kube-api-access-rt4c4\") pod \"02f31482-65f6-4cc8-925e-aaacd7045cf5\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " Jul 20 05:42:31.378733 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378716 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f31482-65f6-4cc8-925e-aaacd7045cf5-istiod-ca-cert\") pod \"02f31482-65f6-4cc8-925e-aaacd7045cf5\" (UID: \"02f31482-65f6-4cc8-925e-aaacd7045cf5\") " Jul 20 05:42:31.379125 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378785 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02f31482-65f6-4cc8-925e-aaacd7045cf5" (UID: "02f31482-65f6-4cc8-925e-aaacd7045cf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:31.379125 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.378858 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02f31482-65f6-4cc8-925e-aaacd7045cf5" (UID: "02f31482-65f6-4cc8-925e-aaacd7045cf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:31.379125 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.379000 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:31.379125 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.379019 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:31.379125 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.379102 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f31482-65f6-4cc8-925e-aaacd7045cf5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02f31482-65f6-4cc8-925e-aaacd7045cf5" (UID: "02f31482-65f6-4cc8-925e-aaacd7045cf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:42:31.379388 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.379186 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02f31482-65f6-4cc8-925e-aaacd7045cf5" (UID: "02f31482-65f6-4cc8-925e-aaacd7045cf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:31.379668 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.379640 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-data" (OuterVolumeSpecName: "istio-data") pod "02f31482-65f6-4cc8-925e-aaacd7045cf5" (UID: "02f31482-65f6-4cc8-925e-aaacd7045cf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:31.380841 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.380819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02f31482-65f6-4cc8-925e-aaacd7045cf5" (UID: "02f31482-65f6-4cc8-925e-aaacd7045cf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:42:31.380950 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.380894 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02f31482-65f6-4cc8-925e-aaacd7045cf5" (UID: "02f31482-65f6-4cc8-925e-aaacd7045cf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:42:31.381267 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.381247 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-token" (OuterVolumeSpecName: "istio-token") pod "02f31482-65f6-4cc8-925e-aaacd7045cf5" (UID: "02f31482-65f6-4cc8-925e-aaacd7045cf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:42:31.381365 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.381321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-kube-api-access-rt4c4" (OuterVolumeSpecName: "kube-api-access-rt4c4") pod "02f31482-65f6-4cc8-925e-aaacd7045cf5" (UID: "02f31482-65f6-4cc8-925e-aaacd7045cf5"). InnerVolumeSpecName "kube-api-access-rt4c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:42:31.479609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.479576 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:31.479609 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.479611 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:31.479901 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.479627 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4c4\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-kube-api-access-rt4c4\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:31.479901 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.479640 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f31482-65f6-4cc8-925e-aaacd7045cf5-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:31.479901 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.479654 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:31.479901 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.479666 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:31.479901 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.479679 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f31482-65f6-4cc8-925e-aaacd7045cf5-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:42:31.902754 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.902716 2579 generic.go:356] "Generic (PLEG): container finished" podID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerID="2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc" exitCode=0 Jul 20 05:42:31.902929 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.902763 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" Jul 20 05:42:31.902929 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.902800 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" event={"ID":"02f31482-65f6-4cc8-925e-aaacd7045cf5","Type":"ContainerDied","Data":"2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc"} Jul 20 05:42:31.902929 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.902855 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n" event={"ID":"02f31482-65f6-4cc8-925e-aaacd7045cf5","Type":"ContainerDied","Data":"c51c744aa807c2e2293c8a339943fc04db8b8ef134e14023689e773b86ecdf4e"} Jul 20 05:42:31.902929 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.902884 2579 scope.go:117] "RemoveContainer" containerID="2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc" Jul 20 05:42:31.912276 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.912260 2579 scope.go:117] "RemoveContainer" containerID="5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d" Jul 20 05:42:31.919583 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.919568 2579 scope.go:117] "RemoveContainer" containerID="2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc" Jul 20 05:42:31.919791 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:42:31.919774 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc\": container with ID starting with 2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc not found: ID does not exist" containerID="2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc" Jul 20 05:42:31.919845 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.919799 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc"} err="failed to get container status \"2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc\": rpc error: code = NotFound desc = could not find container \"2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc\": container with ID starting with 2b8204b8dcb0984730bebec56e6db2a2fbb6ede460da005fcb9de5c0827749dc not found: ID does not exist" Jul 20 05:42:31.919845 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.919813 2579 scope.go:117] "RemoveContainer" containerID="5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d" Jul 20 05:42:31.920081 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:42:31.920045 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d\": container with ID starting with 5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d not found: ID does not exist" containerID="5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d" Jul 20 05:42:31.920174 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.920082 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d"} err="failed to get container status \"5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d\": rpc error: code = NotFound desc = could not find container \"5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d\": container with ID starting with 5401c97488991977bffa0f39d01eb97666aff38cf5f05605fe39b9caa34fa93d not found: ID does not exist" Jul 20 05:42:31.925493 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.925471 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n"] Jul 20 05:42:31.929610 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:31.929587 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d0511a-openshift-default-f9d69577c-p7t9n"] Jul 20 05:42:33.154639 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:33.154607 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" path="/var/lib/kubelet/pods/02f31482-65f6-4cc8-925e-aaacd7045cf5/volumes" Jul 20 05:42:34.892217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.887225 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x"] Jul 20 05:42:34.892217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.888016 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" Jul 20 05:42:34.892217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.888036 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" Jul 20 05:42:34.892217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.888073 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" Jul 20 05:42:34.892217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.888082 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" Jul 20 05:42:34.892217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.888107 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" Jul 20 05:42:34.892217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.888117 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c7167f-5210-4da7-b9ca-fd90a92b372f" containerName="istio-proxy" Jul 20 05:42:34.892217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.888279 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" Jul 20 05:42:34.895473 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.895451 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:34.898148 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.898050 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-82cedd-openshift-default-dockercfg-rvmlt\"" Jul 20 05:42:34.901050 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:34.901029 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x"] Jul 20 05:42:35.008974 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.008937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlwn\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-kube-api-access-xvlwn\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.009181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.008988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df63fa5-8af3-4051-9af2-43261987613e-istiod-ca-cert\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.009181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.009013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-certs\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.009181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.009073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-envoy\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.009181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.009103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df63fa5-8af3-4051-9af2-43261987613e-istio-podinfo\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.009181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.009136 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-credential-socket\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.009181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.009153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-istio-token\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.009401 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.009192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-data\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.009401 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.009210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-socket\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.110539 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-credential-socket\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.110539 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-istio-token\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.110799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-data\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.110799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-socket\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.110799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlwn\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-kube-api-access-xvlwn\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.110975 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df63fa5-8af3-4051-9af2-43261987613e-istiod-ca-cert\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.110975 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-certs\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.110975 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110954 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-data\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.110975 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-envoy\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.111174 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.110979 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-credential-socket\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.111174 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.111024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-socket\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.111174 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.111037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df63fa5-8af3-4051-9af2-43261987613e-istio-podinfo\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.111273 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.111224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-certs\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.111490 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.111469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df63fa5-8af3-4051-9af2-43261987613e-istiod-ca-cert\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.113165 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.113134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-envoy\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.113357 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.113338 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df63fa5-8af3-4051-9af2-43261987613e-istio-podinfo\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.118200 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.118163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-istio-token\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.118313 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.118296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlwn\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-kube-api-access-xvlwn\") pod \"e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.206916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.206836 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:35.345097 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.344719 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x"] Jul 20 05:42:35.347928 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:42:35.347900 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df63fa5_8af3_4051_9af2_43261987613e.slice/crio-fc10b0ea22da1280860da90e602517d4c24ef9311e06c684fd857eaa73b26271 WatchSource:0}: Error finding container fc10b0ea22da1280860da90e602517d4c24ef9311e06c684fd857eaa73b26271: Status 404 returned error can't find the container with id fc10b0ea22da1280860da90e602517d4c24ef9311e06c684fd857eaa73b26271 Jul 20 05:42:35.350984 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.350953 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:35.351069 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.351017 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:35.351069 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.351045 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:35.923319 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.923286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" event={"ID":"4df63fa5-8af3-4051-9af2-43261987613e","Type":"ContainerStarted","Data":"ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e"} Jul 20 05:42:35.923789 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.923334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" event={"ID":"4df63fa5-8af3-4051-9af2-43261987613e","Type":"ContainerStarted","Data":"fc10b0ea22da1280860da90e602517d4c24ef9311e06c684fd857eaa73b26271"} Jul 20 05:42:35.943262 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:35.943213 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" podStartSLOduration=1.943199469 podStartE2EDuration="1.943199469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 05:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:42:35.942463299 +0000 UTC m=+2585.211946983" watchObservedRunningTime="2026-07-20 05:42:35.943199469 +0000 UTC m=+2585.212683154" Jul 20 05:42:36.207945 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:36.207854 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:36.212542 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:36.212516 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:36.928071 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:36.928031 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:36.928852 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:36.928832 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:37.990277 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:37.990246 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:42:44.929218 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:44.929181 2579 patch_prober.go:28] interesting pod/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 20 05:42:44.929645 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:44.929253 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 20 05:42:45.975275 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:45.975180 2579 generic.go:356] "Generic (PLEG): container finished" podID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerID="b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47" exitCode=0 Jul 20 05:42:45.975943 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:45.975914 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" event={"ID":"7de49bff-8ae4-45ca-b7fd-5c3557211c65","Type":"ContainerDied","Data":"b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47"} Jul 20 05:42:45.976563 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:45.976545 2579 scope.go:117] "RemoveContainer" containerID="b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47" Jul 20 05:42:45.978716 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:45.978679 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:45.978919 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:45.978895 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:45.979100 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:45.979078 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:46.985932 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:46.985892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" event={"ID":"7de49bff-8ae4-45ca-b7fd-5c3557211c65","Type":"ContainerStarted","Data":"1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119"} Jul 20 05:42:47.224779 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:47.224746 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:47.224982 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:47.224957 2579 patch_prober.go:28] interesting pod/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 20 05:42:47.225089 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:47.225029 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 20 05:42:48.225073 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:48.225017 2579 patch_prober.go:28] interesting pod/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 20 05:42:48.225451 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:48.225095 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 20 05:42:49.225513 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:49.225467 2579 patch_prober.go:28] interesting pod/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 20 05:42:49.225927 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:49.225526 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 20 05:42:50.225723 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:50.225629 2579 patch_prober.go:28] interesting pod/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 20 05:42:50.225723 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:50.225697 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 20 05:42:51.229673 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:51.229644 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:51.230114 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:51.229932 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:51.230680 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:51.230657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:42:51.618364 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:51.618321 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:42:52.970233 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:52.970185 2579 patch_prober.go:28] interesting pod/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 20 05:42:52.972619 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:52.970276 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 20 05:42:54.025310 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:54.025277 2579 generic.go:356] "Generic (PLEG): container finished" podID="4df63fa5-8af3-4051-9af2-43261987613e" containerID="ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e" exitCode=0 Jul 20 05:42:54.025792 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:54.025357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" event={"ID":"4df63fa5-8af3-4051-9af2-43261987613e","Type":"ContainerDied","Data":"ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e"} Jul 20 05:42:54.025860 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:54.025849 2579 scope.go:117] "RemoveContainer" containerID="ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e" Jul 20 05:42:54.026859 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:54.026832 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:54.026916 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:54.026897 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:54.026952 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:54.026932 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 20 05:42:55.038620 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:55.038586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" event={"ID":"4df63fa5-8af3-4051-9af2-43261987613e","Type":"ContainerStarted","Data":"daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9"} Jul 20 05:42:55.207244 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:55.207204 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:55.207419 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:55.207385 2579 patch_prober.go:28] interesting pod/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 20 05:42:55.207465 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:55.207433 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 20 05:42:56.207572 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:56.207536 2579 patch_prober.go:28] interesting pod/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 20 05:42:56.207972 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:56.207595 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 20 05:42:57.208212 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:57.208176 2579 patch_prober.go:28] interesting pod/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 20 05:42:57.208642 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:57.208241 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 20 05:42:58.207496 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:58.207457 2579 patch_prober.go:28] interesting pod/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 20 05:42:58.207689 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:58.207522 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 20 05:42:59.211023 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:59.210991 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:59.211517 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:59.211393 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:42:59.211948 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:42:59.211928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:43:01.422845 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:43:01.422811 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:43:27.613208 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:43:27.613174 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:43:43.622685 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:43:43.622650 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:44:02.023149 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:44:02.023110 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:44:17.225796 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:44:17.225755 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:44:36.230986 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:44:36.230949 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:44:56.323651 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:44:56.323615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:45:15.917129 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:45:15.917093 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:45:34.617672 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:45:34.617637 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:45:51.127078 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:45:51.127027 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:46:33.722569 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:46:33.722525 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:46:48.718092 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:46:48.718043 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:47:10.422024 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:47:10.421990 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:47:26.628411 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:47:26.628369 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:47:40.719751 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:47:40.719708 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:47:54.728510 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:47:54.728473 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:48:09.721155 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:48:09.721124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:48:24.028130 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:48:24.028090 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:48:49.920003 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:48:49.919966 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:48:58.295906 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:48:58.295825 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2"] Jul 20 05:48:58.296516 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:48:58.296126 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" containerID="cri-o://1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119" gracePeriod=30 Jul 20 05:49:03.550747 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.550722 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:49:03.585631 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585569 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-certs\") pod \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " Jul 20 05:49:03.585631 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585618 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istiod-ca-cert\") pod \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " Jul 20 05:49:03.585806 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585641 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-token\") pod \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " Jul 20 05:49:03.585806 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr99l\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-kube-api-access-dr99l\") pod \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " Jul 20 05:49:03.585806 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585705 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-credential-socket\") pod \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " Jul 20 05:49:03.585806 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585729 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-socket\") pod \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " Jul 20 05:49:03.585806 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585755 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-podinfo\") pod \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " Jul 20 05:49:03.585806 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585801 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-data\") pod \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " Jul 20 05:49:03.586130 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585835 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-envoy\") pod \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\" (UID: \"7de49bff-8ae4-45ca-b7fd-5c3557211c65\") " Jul 20 05:49:03.586130 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.585853 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7de49bff-8ae4-45ca-b7fd-5c3557211c65" (UID: "7de49bff-8ae4-45ca-b7fd-5c3557211c65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:03.586130 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.586014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7de49bff-8ae4-45ca-b7fd-5c3557211c65" (UID: "7de49bff-8ae4-45ca-b7fd-5c3557211c65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:49:03.586130 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.586047 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7de49bff-8ae4-45ca-b7fd-5c3557211c65" (UID: "7de49bff-8ae4-45ca-b7fd-5c3557211c65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:03.586130 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.586126 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:03.586410 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.586144 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:03.586410 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.586156 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:03.586410 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.586272 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7de49bff-8ae4-45ca-b7fd-5c3557211c65" (UID: "7de49bff-8ae4-45ca-b7fd-5c3557211c65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:03.587034 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.587001 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-data" (OuterVolumeSpecName: "istio-data") pod "7de49bff-8ae4-45ca-b7fd-5c3557211c65" (UID: "7de49bff-8ae4-45ca-b7fd-5c3557211c65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:03.588216 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.588189 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-kube-api-access-dr99l" (OuterVolumeSpecName: "kube-api-access-dr99l") pod "7de49bff-8ae4-45ca-b7fd-5c3557211c65" (UID: "7de49bff-8ae4-45ca-b7fd-5c3557211c65"). InnerVolumeSpecName "kube-api-access-dr99l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:49:03.588301 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.588287 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7de49bff-8ae4-45ca-b7fd-5c3557211c65" (UID: "7de49bff-8ae4-45ca-b7fd-5c3557211c65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:03.588410 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.588390 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7de49bff-8ae4-45ca-b7fd-5c3557211c65" (UID: "7de49bff-8ae4-45ca-b7fd-5c3557211c65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:49:03.588467 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.588401 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-token" (OuterVolumeSpecName: "istio-token") pod "7de49bff-8ae4-45ca-b7fd-5c3557211c65" (UID: "7de49bff-8ae4-45ca-b7fd-5c3557211c65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:49:03.686579 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.686548 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:03.686579 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.686576 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:03.686579 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.686584 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:03.686823 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.686592 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr99l\" (UniqueName: \"kubernetes.io/projected/7de49bff-8ae4-45ca-b7fd-5c3557211c65-kube-api-access-dr99l\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:03.686823 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.686601 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de49bff-8ae4-45ca-b7fd-5c3557211c65-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:03.686823 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.686611 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de49bff-8ae4-45ca-b7fd-5c3557211c65-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:03.760997 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.760967 2579 generic.go:356] "Generic (PLEG): container finished" podID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerID="1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119" exitCode=0 Jul 20 05:49:03.761150 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.761015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" event={"ID":"7de49bff-8ae4-45ca-b7fd-5c3557211c65","Type":"ContainerDied","Data":"1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119"} Jul 20 05:49:03.761150 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.761079 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" event={"ID":"7de49bff-8ae4-45ca-b7fd-5c3557211c65","Type":"ContainerDied","Data":"1f0483365b64f32138557b819bc5a057b31bb0049fb42efde4e75517622c82d0"} Jul 20 05:49:03.761150 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.761088 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2" Jul 20 05:49:03.761150 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.761102 2579 scope.go:117] "RemoveContainer" containerID="1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119" Jul 20 05:49:03.770544 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.770525 2579 scope.go:117] "RemoveContainer" containerID="b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47" Jul 20 05:49:03.777856 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.777836 2579 scope.go:117] "RemoveContainer" containerID="1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119" Jul 20 05:49:03.778107 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:49:03.778087 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119\": container with ID starting with 1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119 not found: ID does not exist" containerID="1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119" Jul 20 05:49:03.778171 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.778114 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119"} err="failed to get container status \"1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119\": rpc error: code = NotFound desc = could not find container \"1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119\": container with ID starting with 1d0e772025ea4af9f29e6a96a99fec058106a11d01027bc60074c7ab83fdf119 not found: ID does not exist" Jul 20 05:49:03.778171 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.778128 2579 scope.go:117] "RemoveContainer" containerID="b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47" Jul 20 05:49:03.778373 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:49:03.778353 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47\": container with ID starting with b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47 not found: ID does not exist" containerID="b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47" Jul 20 05:49:03.778432 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.778380 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47"} err="failed to get container status \"b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47\": rpc error: code = NotFound desc = could not find container \"b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47\": container with ID starting with b9985b35803b7f360af8375f16cc246a9e4715b3267dcd70413e44e8b5fbcf47 not found: ID does not exist" Jul 20 05:49:03.783539 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.783494 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2"] Jul 20 05:49:03.785258 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:03.785238 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-851aef-openshift-default-69f7799847-2twb2"] Jul 20 05:49:05.155289 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:05.155255 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" path="/var/lib/kubelet/pods/7de49bff-8ae4-45ca-b7fd-5c3557211c65/volumes" Jul 20 05:49:12.222479 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:12.222441 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dm8sm"] Jul 20 05:49:20.327081 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:20.327014 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x"] Jul 20 05:49:20.327549 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:20.327300 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" containerID="cri-o://daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9" gracePeriod=30 Jul 20 05:49:25.483945 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.483913 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:49:25.542594 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542553 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-data\") pod \"4df63fa5-8af3-4051-9af2-43261987613e\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " Jul 20 05:49:25.542787 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-credential-socket\") pod \"4df63fa5-8af3-4051-9af2-43261987613e\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " Jul 20 05:49:25.542787 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542624 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-certs\") pod \"4df63fa5-8af3-4051-9af2-43261987613e\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " Jul 20 05:49:25.542787 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542641 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-socket\") pod \"4df63fa5-8af3-4051-9af2-43261987613e\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " Jul 20 05:49:25.542787 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542662 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df63fa5-8af3-4051-9af2-43261987613e-istiod-ca-cert\") pod \"4df63fa5-8af3-4051-9af2-43261987613e\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " Jul 20 05:49:25.542787 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542684 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvlwn\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-kube-api-access-xvlwn\") pod \"4df63fa5-8af3-4051-9af2-43261987613e\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " Jul 20 05:49:25.542787 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542717 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-envoy\") pod \"4df63fa5-8af3-4051-9af2-43261987613e\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " Jul 20 05:49:25.543114 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542887 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df63fa5-8af3-4051-9af2-43261987613e-istio-podinfo\") pod \"4df63fa5-8af3-4051-9af2-43261987613e\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " Jul 20 05:49:25.543114 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542888 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4df63fa5-8af3-4051-9af2-43261987613e" (UID: "4df63fa5-8af3-4051-9af2-43261987613e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:25.543114 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-istio-token\") pod \"4df63fa5-8af3-4051-9af2-43261987613e\" (UID: \"4df63fa5-8af3-4051-9af2-43261987613e\") " Jul 20 05:49:25.543114 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.542952 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4df63fa5-8af3-4051-9af2-43261987613e" (UID: "4df63fa5-8af3-4051-9af2-43261987613e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:25.543114 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.543017 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4df63fa5-8af3-4051-9af2-43261987613e" (UID: "4df63fa5-8af3-4051-9af2-43261987613e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:25.543304 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.543153 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df63fa5-8af3-4051-9af2-43261987613e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4df63fa5-8af3-4051-9af2-43261987613e" (UID: "4df63fa5-8af3-4051-9af2-43261987613e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 20 05:49:25.543304 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.543245 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-credential-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:25.543304 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.543263 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-certs\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:25.543304 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.543272 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-workload-socket\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:25.543304 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.543280 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4df63fa5-8af3-4051-9af2-43261987613e-istiod-ca-cert\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:25.543536 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.543466 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4df63fa5-8af3-4051-9af2-43261987613e" (UID: "4df63fa5-8af3-4051-9af2-43261987613e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:25.545075 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.545030 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4df63fa5-8af3-4051-9af2-43261987613e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4df63fa5-8af3-4051-9af2-43261987613e" (UID: "4df63fa5-8af3-4051-9af2-43261987613e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 20 05:49:25.545190 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.545076 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4df63fa5-8af3-4051-9af2-43261987613e" (UID: "4df63fa5-8af3-4051-9af2-43261987613e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 20 05:49:25.545190 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.545128 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4df63fa5-8af3-4051-9af2-43261987613e" (UID: "4df63fa5-8af3-4051-9af2-43261987613e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:49:25.545292 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.545235 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-kube-api-access-xvlwn" (OuterVolumeSpecName: "kube-api-access-xvlwn") pod "4df63fa5-8af3-4051-9af2-43261987613e" (UID: "4df63fa5-8af3-4051-9af2-43261987613e"). InnerVolumeSpecName "kube-api-access-xvlwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 20 05:49:25.644447 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.644363 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-data\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:25.644447 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.644394 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlwn\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-kube-api-access-xvlwn\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:25.644447 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.644404 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4df63fa5-8af3-4051-9af2-43261987613e-istio-envoy\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:25.644447 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.644412 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4df63fa5-8af3-4051-9af2-43261987613e-istio-podinfo\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:25.644447 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.644420 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4df63fa5-8af3-4051-9af2-43261987613e-istio-token\") on node \"ip-10-0-133-196.ec2.internal\" DevicePath \"\"" Jul 20 05:49:25.862254 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.862218 2579 generic.go:356] "Generic (PLEG): container finished" podID="4df63fa5-8af3-4051-9af2-43261987613e" containerID="daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9" exitCode=0 Jul 20 05:49:25.862443 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.862261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" event={"ID":"4df63fa5-8af3-4051-9af2-43261987613e","Type":"ContainerDied","Data":"daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9"} Jul 20 05:49:25.862443 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.862308 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" event={"ID":"4df63fa5-8af3-4051-9af2-43261987613e","Type":"ContainerDied","Data":"fc10b0ea22da1280860da90e602517d4c24ef9311e06c684fd857eaa73b26271"} Jul 20 05:49:25.862443 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.862337 2579 scope.go:117] "RemoveContainer" containerID="daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9" Jul 20 05:49:25.862443 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.862264 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x" Jul 20 05:49:25.871039 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.871022 2579 scope.go:117] "RemoveContainer" containerID="ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e" Jul 20 05:49:25.878959 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.878939 2579 scope.go:117] "RemoveContainer" containerID="daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9" Jul 20 05:49:25.879239 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:49:25.879220 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9\": container with ID starting with daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9 not found: ID does not exist" containerID="daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9" Jul 20 05:49:25.879289 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.879247 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9"} err="failed to get container status \"daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9\": rpc error: code = NotFound desc = could not find container \"daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9\": container with ID starting with daf6752ef13fdecad65ff83205f4efeba3fe599e1ebff4bf1be077e1717b20b9 not found: ID does not exist" Jul 20 05:49:25.879289 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.879265 2579 scope.go:117] "RemoveContainer" containerID="ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e" Jul 20 05:49:25.879508 ip-10-0-133-196 kubenswrapper[2579]: E0720 05:49:25.879491 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e\": container with ID starting with ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e not found: ID does not exist" containerID="ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e" Jul 20 05:49:25.879546 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.879514 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e"} err="failed to get container status \"ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e\": rpc error: code = NotFound desc = could not find container \"ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e\": container with ID starting with ebe085aeb44f182bc67b526babb2325453643487aed85322549866548b3d698e not found: ID does not exist" Jul 20 05:49:25.884243 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.884221 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x"] Jul 20 05:49:25.890107 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:25.890085 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-82cedd-openshift-default-857dfdb4cf-bf54x"] Jul 20 05:49:26.665871 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:26.665841 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bgkb4_a6e3649c-b5e4-451d-936c-b47e8baaeb2e/manager/0.log" Jul 20 05:49:26.787253 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:26.787222 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gnfcq_33b52d2a-b130-4f0f-b80c-7f698cb71ec7/manager/0.log" Jul 20 05:49:27.025524 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:27.025496 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vxr6x_8619931b-64ab-4871-9ff8-314504bb263a/server/0.log" Jul 20 05:49:27.149513 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:27.149485 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pqmcj_a6e0da71-dac3-475e-9fa6-5df2be58f7f1/manager/0.log" Jul 20 05:49:27.154384 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:27.154354 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df63fa5-8af3-4051-9af2-43261987613e" path="/var/lib/kubelet/pods/4df63fa5-8af3-4051-9af2-43261987613e/volumes" Jul 20 05:49:27.393576 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:27.393495 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54dff8fcf7-kzrfv_91136ba1-8512-40ca-8958-1f7b33aed995/copy-manifests/0.log" Jul 20 05:49:27.407536 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:27.407508 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54dff8fcf7-kzrfv_91136ba1-8512-40ca-8958-1f7b33aed995/manager/0.log" Jul 20 05:49:30.348427 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:30.348390 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-5bsx2_732b8b95-40f0-472f-aada-5e5e16fddb82/manager/0.log" Jul 20 05:49:30.491605 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:30.491573 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dm8sm_12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8/limitador/0.log" Jul 20 05:49:31.108983 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:31.108949 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wxx59_b02c8f0d-0e3a-4755-9aff-17c46c29b47a/discovery/0.log" Jul 20 05:49:31.340025 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:31.339992 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bf9656866-lwmqw_87c4c318-4559-4148-ae06-c2a400fbe6b3/kube-auth-proxy/0.log" Jul 20 05:49:32.668077 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:32.668030 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-pxcr2_bc245fd5-991a-4d29-9918-8fe3f5ad18be/main/0.log" Jul 20 05:49:41.149369 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:41.149335 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c7ls5_bbc9f1d6-06c2-4142-bf58-7b25b80927d4/global-pull-secret-syncer/0.log" Jul 20 05:49:41.333268 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:41.333237 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fpdbr_96dcb202-5818-45a8-a772-b148517779f2/konnectivity-agent/0.log" Jul 20 05:49:41.376105 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:41.376075 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-196.ec2.internal_94f9424feb20d8f7b53d0fbc4a5441ee/haproxy/0.log" Jul 20 05:49:46.888081 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:46.888030 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-5bsx2_732b8b95-40f0-472f-aada-5e5e16fddb82/manager/0.log" Jul 20 05:49:46.916799 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:46.916767 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dm8sm_12ab696c-9f2b-44e9-a6c8-e6cd67bd74c8/limitador/0.log" Jul 20 05:49:48.657724 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:48.657658 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac3278cd-1342-427d-80a2-25d62c461f3b/alertmanager/0.log" Jul 20 05:49:48.678941 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:48.678918 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac3278cd-1342-427d-80a2-25d62c461f3b/config-reloader/0.log" Jul 20 05:49:48.698403 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:48.698382 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac3278cd-1342-427d-80a2-25d62c461f3b/kube-rbac-proxy-web/0.log" Jul 20 05:49:48.720776 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:48.720761 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac3278cd-1342-427d-80a2-25d62c461f3b/kube-rbac-proxy/0.log" Jul 20 05:49:48.742773 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:48.742754 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac3278cd-1342-427d-80a2-25d62c461f3b/kube-rbac-proxy-metric/0.log" Jul 20 05:49:48.762702 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:48.762675 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac3278cd-1342-427d-80a2-25d62c461f3b/prom-label-proxy/0.log" Jul 20 05:49:48.783263 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:48.783244 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ac3278cd-1342-427d-80a2-25d62c461f3b/init-config-reloader/0.log" Jul 20 05:49:48.942095 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:48.941989 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-p4t9s_b92cdf5c-6254-4b35-bb4a-170cb1971ad7/monitoring-plugin/0.log" Jul 20 05:49:48.982211 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:48.982169 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gh5fb_b04d0856-7c69-4200-bf79-ac7bab49f604/node-exporter/0.log" Jul 20 05:49:49.002701 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:49.002677 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gh5fb_b04d0856-7c69-4200-bf79-ac7bab49f604/kube-rbac-proxy/0.log" Jul 20 05:49:49.022667 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:49.022641 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gh5fb_b04d0856-7c69-4200-bf79-ac7bab49f604/init-textfile/0.log" Jul 20 05:49:49.496820 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:49.496748 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-6t828_8e17150b-521d-4acd-b809-7fad7c504398/prometheus-operator-admission-webhook/0.log" Jul 20 05:49:51.292715 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.292676 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42"] Jul 20 05:49:51.293212 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293166 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" Jul 20 05:49:51.293212 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293186 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" Jul 20 05:49:51.293212 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293197 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" Jul 20 05:49:51.293212 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293207 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" Jul 20 05:49:51.293436 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293228 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" Jul 20 05:49:51.293436 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293237 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" Jul 20 05:49:51.293436 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293319 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" Jul 20 05:49:51.293436 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293329 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" Jul 20 05:49:51.293436 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293338 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df63fa5-8af3-4051-9af2-43261987613e" containerName="istio-proxy" Jul 20 05:49:51.293436 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293347 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de49bff-8ae4-45ca-b7fd-5c3557211c65" containerName="istio-proxy" Jul 20 05:49:51.293436 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.293358 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="02f31482-65f6-4cc8-925e-aaacd7045cf5" containerName="istio-proxy" Jul 20 05:49:51.296487 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.296464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.299181 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.299157 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g8fln\"/\"openshift-service-ca.crt\"" Jul 20 05:49:51.300278 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.300253 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g8fln\"/\"kube-root-ca.crt\"" Jul 20 05:49:51.300375 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.300263 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g8fln\"/\"default-dockercfg-c52p4\"" Jul 20 05:49:51.306539 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.306513 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42"] Jul 20 05:49:51.364983 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.364948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-proc\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.364983 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.364981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5d55\" (UniqueName: \"kubernetes.io/projected/d3e96647-f025-407c-8e2c-23266d37fea1-kube-api-access-j5d55\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.365217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.365016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-sys\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.365217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.365128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-podres\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.365217 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.365150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-lib-modules\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.465763 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.465728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-podres\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.465763 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.465765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-lib-modules\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.466002 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.465787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-proc\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.466002 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.465837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5d55\" (UniqueName: \"kubernetes.io/projected/d3e96647-f025-407c-8e2c-23266d37fea1-kube-api-access-j5d55\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.466002 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.465884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-lib-modules\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.466002 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.465886 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-proc\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.466002 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.465885 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-podres\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.466002 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.465897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-sys\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.466002 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.465946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3e96647-f025-407c-8e2c-23266d37fea1-sys\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.473930 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.473907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5d55\" (UniqueName: \"kubernetes.io/projected/d3e96647-f025-407c-8e2c-23266d37fea1-kube-api-access-j5d55\") pod \"perf-node-gather-daemonset-xkj42\" (UID: \"d3e96647-f025-407c-8e2c-23266d37fea1\") " pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.606792 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.606700 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:51.733583 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.733402 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42"] Jul 20 05:49:51.736420 ip-10-0-133-196 kubenswrapper[2579]: W0720 05:49:51.736392 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd3e96647_f025_407c_8e2c_23266d37fea1.slice/crio-b38775a484138f40c6d98b12fd015c06bf0d6eb5d9c93ce0429edbf0e7eebc73 WatchSource:0}: Error finding container b38775a484138f40c6d98b12fd015c06bf0d6eb5d9c93ce0429edbf0e7eebc73: Status 404 returned error can't find the container with id b38775a484138f40c6d98b12fd015c06bf0d6eb5d9c93ce0429edbf0e7eebc73 Jul 20 05:49:51.986401 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.986365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" event={"ID":"d3e96647-f025-407c-8e2c-23266d37fea1","Type":"ContainerStarted","Data":"1bc266c919f2a84bb371bccdd57970a12457f00ee1683d0213efc9cbcc3bce3b"} Jul 20 05:49:51.986612 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.986421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" event={"ID":"d3e96647-f025-407c-8e2c-23266d37fea1","Type":"ContainerStarted","Data":"b38775a484138f40c6d98b12fd015c06bf0d6eb5d9c93ce0429edbf0e7eebc73"} Jul 20 05:49:51.986612 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:51.986486 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:52.007024 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:52.006968 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" podStartSLOduration=1.006953863 podStartE2EDuration="1.006953863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 05:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 05:49:52.006726919 +0000 UTC m=+3021.276210605" watchObservedRunningTime="2026-07-20 05:49:52.006953863 +0000 UTC m=+3021.276437548" Jul 20 05:49:53.578834 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:53.578803 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-95875_87705952-f6c4-4c65-83a7-1a04256c34b1/dns/0.log" Jul 20 05:49:53.600801 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:53.600760 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-95875_87705952-f6c4-4c65-83a7-1a04256c34b1/kube-rbac-proxy/0.log" Jul 20 05:49:53.764386 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:53.764348 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jv8n9_825150de-fdc4-4920-a483-b648d894dffc/dns-node-resolver/0.log" Jul 20 05:49:54.297227 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:54.297197 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h2rbz_90688000-32fe-4a0b-8dc3-32e98e1fd32c/node-ca/0.log" Jul 20 05:49:55.351023 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:55.350988 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wxx59_b02c8f0d-0e3a-4755-9aff-17c46c29b47a/discovery/0.log" Jul 20 05:49:55.392946 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:55.392917 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bf9656866-lwmqw_87c4c318-4559-4148-ae06-c2a400fbe6b3/kube-auth-proxy/0.log" Jul 20 05:49:56.218307 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:56.218273 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m59z4_7ff43437-bb38-4b62-93e7-a9cd4cbc30d6/serve-healthcheck-canary/0.log" Jul 20 05:49:56.791124 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:56.791094 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxxv5_6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01/kube-rbac-proxy/0.log" Jul 20 05:49:56.811360 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:56.811339 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxxv5_6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01/exporter/0.log" Jul 20 05:49:56.830806 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:56.830786 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxxv5_6fdcdf91-bb28-42b4-9d0d-d8eb6ed02c01/extractor/0.log" Jul 20 05:49:58.001656 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:58.001631 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g8fln/perf-node-gather-daemonset-xkj42" Jul 20 05:49:58.952651 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:58.952625 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bgkb4_a6e3649c-b5e4-451d-936c-b47e8baaeb2e/manager/0.log" Jul 20 05:49:59.142400 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:59.142369 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gnfcq_33b52d2a-b130-4f0f-b80c-7f698cb71ec7/manager/0.log" Jul 20 05:49:59.309187 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:59.309160 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vxr6x_8619931b-64ab-4871-9ff8-314504bb263a/server/0.log" Jul 20 05:49:59.384167 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:59.384136 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-pqmcj_a6e0da71-dac3-475e-9fa6-5df2be58f7f1/manager/0.log" Jul 20 05:49:59.496029 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:59.496000 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54dff8fcf7-kzrfv_91136ba1-8512-40ca-8958-1f7b33aed995/manager/0.log" Jul 20 05:49:59.511193 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:49:59.511168 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54dff8fcf7-kzrfv_91136ba1-8512-40ca-8958-1f7b33aed995/copy-manifests/0.log" Jul 20 05:50:00.997904 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:00.997874 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-rhnd2_0f6ad10f-f268-4c52-b51a-4b7032f382c1/openshift-lws-operator/0.log" Jul 20 05:50:01.938370 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:01.938341 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wxx59_b02c8f0d-0e3a-4755-9aff-17c46c29b47a/discovery/0.log" Jul 20 05:50:01.961371 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:01.961347 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bf9656866-lwmqw_87c4c318-4559-4148-ae06-c2a400fbe6b3/kube-auth-proxy/0.log" Jul 20 05:50:03.417722 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:03.417695 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-wxx59_b02c8f0d-0e3a-4755-9aff-17c46c29b47a/discovery/0.log" Jul 20 05:50:03.447257 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:03.447232 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bf9656866-lwmqw_87c4c318-4559-4148-ae06-c2a400fbe6b3/kube-auth-proxy/0.log" Jul 20 05:50:07.404158 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.404080 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnfx_5fa014da-b395-4e6a-8804-5d3217305e99/kube-multus-additional-cni-plugins/0.log" Jul 20 05:50:07.427154 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.427128 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnfx_5fa014da-b395-4e6a-8804-5d3217305e99/egress-router-binary-copy/0.log" Jul 20 05:50:07.446849 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.446831 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnfx_5fa014da-b395-4e6a-8804-5d3217305e99/cni-plugins/0.log" Jul 20 05:50:07.466647 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.466623 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnfx_5fa014da-b395-4e6a-8804-5d3217305e99/bond-cni-plugin/0.log" Jul 20 05:50:07.486830 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.486800 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnfx_5fa014da-b395-4e6a-8804-5d3217305e99/routeoverride-cni/0.log" Jul 20 05:50:07.506351 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.506331 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnfx_5fa014da-b395-4e6a-8804-5d3217305e99/whereabouts-cni-bincopy/0.log" Jul 20 05:50:07.525771 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.525752 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5gnfx_5fa014da-b395-4e6a-8804-5d3217305e99/whereabouts-cni/0.log" Jul 20 05:50:07.905338 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.905311 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vt4vk_21f9b4ba-dead-455b-ac3f-78d65f54ef30/kube-multus/0.log" Jul 20 05:50:07.964264 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.964235 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-krtbd_8e5b3021-37f7-4153-bb80-758fb934948c/network-metrics-daemon/0.log" Jul 20 05:50:07.984595 ip-10-0-133-196 kubenswrapper[2579]: I0720 05:50:07.984568 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-krtbd_8e5b3021-37f7-4153-bb80-758fb934948c/kube-rbac-proxy/0.log"