Jul 21 04:07:25.126009 ip-10-0-137-226 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 04:07:25.126019 ip-10-0-137-226 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 04:07:25.126026 ip-10-0-137-226 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 04:07:25.126286 ip-10-0-137-226 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 04:07:35.268459 ip-10-0-137-226 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 04:07:35.268479 ip-10-0-137-226 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 967047b54bd04c06bbdb6db39ea00679 -- Jul 21 04:09:13.373803 ip-10-0-137-226 systemd[1]: Starting Kubernetes Kubelet... Jul 21 04:09:13.799112 ip-10-0-137-226 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 04:09:13.799112 ip-10-0-137-226 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 04:09:13.799112 ip-10-0-137-226 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 04:09:13.799112 ip-10-0-137-226 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 04:09:13.799666 ip-10-0-137-226 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 04:09:13.800201 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.800114 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 04:09:13.806702 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806688 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 04:09:13.806702 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806702 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806706 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806710 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806713 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806716 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806727 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806730 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806733 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806735 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806738 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806742 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806745 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806747 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806750 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806753 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806755 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806758 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806761 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806764 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 04:09:13.806765 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806766 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806774 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806777 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806780 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806783 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806785 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806788 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806790 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806793 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806796 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806798 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806801 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806803 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806805 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806808 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806810 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806813 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806815 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806820 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 04:09:13.807218 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806823 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806826 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806829 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806832 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806835 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806838 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806841 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806843 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806846 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806848 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806851 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806853 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806857 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806860 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806863 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806875 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806879 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806881 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806884 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 04:09:13.807669 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806886 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806889 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806891 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806894 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806897 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806899 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806902 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806904 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806907 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806909 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806912 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806915 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806917 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806920 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806922 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806924 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806927 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806930 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806933 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806935 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 04:09:13.808151 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806938 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806941 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806944 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806946 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806949 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806951 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806954 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806956 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806959 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806969 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806973 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806976 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806978 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806981 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806983 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806986 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806988 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806991 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806993 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806996 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 04:09:13.808637 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.806999 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807712 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807719 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807722 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807725 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807728 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807731 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807733 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807736 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807739 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807741 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807743 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807746 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807749 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807751 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807754 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807756 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807759 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807763 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807767 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 04:09:13.809104 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807770 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807773 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807776 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807778 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807781 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807783 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807786 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807789 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807791 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807794 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807796 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807799 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807801 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807804 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807806 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807809 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807811 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807813 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807816 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807818 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 04:09:13.809600 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807821 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807823 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807826 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807829 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807831 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807834 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807836 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807839 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807841 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807844 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807846 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807849 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807852 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807855 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807857 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807860 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807863 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807867 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807870 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807873 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 04:09:13.810077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807875 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807878 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807881 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807883 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807886 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807889 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807892 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807894 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807897 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807899 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807902 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807905 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807907 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807910 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807912 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807916 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807918 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807921 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807923 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807925 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 04:09:13.810608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807928 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807930 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807933 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807935 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807939 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807942 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807945 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807947 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807950 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807956 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807959 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807962 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807965 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807968 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807970 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807973 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807977 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807980 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807983 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 04:09:13.811242 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.807986 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808389 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808395 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808398 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808401 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808404 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808407 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808410 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808414 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808416 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808420 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808422 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808425 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808428 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808431 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808433 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808436 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808440 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808443 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808446 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 04:09:13.811724 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808449 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808459 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808462 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808464 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808467 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808470 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808472 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808474 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808477 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808495 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808498 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808500 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808503 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808505 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808508 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808511 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808513 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808516 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808518 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808521 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 04:09:13.812230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808524 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808526 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808529 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808531 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808534 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808538 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808541 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808545 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808548 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808552 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808555 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808558 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808561 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808564 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808567 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808570 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808572 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808575 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808577 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 04:09:13.812718 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808580 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808582 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808585 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808587 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808590 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808592 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808595 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808599 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808602 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808605 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808608 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808611 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808614 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808616 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808619 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808622 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808625 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808628 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808632 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808634 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 04:09:13.813165 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808637 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808640 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808645 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808648 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808651 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808654 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808657 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808659 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808662 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808665 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808668 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808670 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808673 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808675 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808678 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808680 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808684 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808687 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808691 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 04:09:13.813686 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808693 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.808696 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808763 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808770 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808776 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808780 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808784 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808788 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808792 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808796 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808799 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808802 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808805 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808809 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808812 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808817 2567 flags.go:64] FLAG: --cgroup-root="" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808820 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808823 2567 flags.go:64] FLAG: --client-ca-file="" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808826 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808829 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808832 2567 flags.go:64] FLAG: --cluster-domain="" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808835 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808839 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808842 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 04:09:13.814152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808846 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808849 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808852 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808855 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808858 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808861 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808864 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808867 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808870 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808873 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808877 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808880 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808883 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808886 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808889 2567 flags.go:64] FLAG: --enable-server="true" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808892 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808896 2567 flags.go:64] FLAG: --event-burst="100" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808900 2567 flags.go:64] FLAG: --event-qps="50" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808905 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808908 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808911 2567 flags.go:64] FLAG: --eviction-hard="" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808915 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808918 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808923 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808926 2567 flags.go:64] FLAG: --eviction-soft="" Jul 21 04:09:13.814793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808929 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808932 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808935 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808938 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808941 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808944 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808947 2567 flags.go:64] FLAG: --feature-gates="" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808951 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808954 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808957 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808960 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808963 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808966 2567 flags.go:64] FLAG: --help="false" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808969 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-137-226.ec2.internal" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808972 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808975 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808978 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808982 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808985 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808988 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808991 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808994 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808996 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 04:09:13.815382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.808999 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809002 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809007 2567 flags.go:64] FLAG: --kube-reserved="" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809010 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809013 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809016 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809018 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809023 2567 flags.go:64] FLAG: --lock-file="" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809026 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809029 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809032 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809037 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809040 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809050 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809053 2567 flags.go:64] FLAG: --logging-format="text" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809056 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809060 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809063 2567 flags.go:64] FLAG: --manifest-url="" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809066 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809070 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809074 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809078 2567 flags.go:64] FLAG: --max-pods="110" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809082 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809085 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809088 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 04:09:13.816004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809091 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809094 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809097 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809100 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809107 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809110 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809113 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809117 2567 flags.go:64] FLAG: --pod-cidr="" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809120 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809144 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809147 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809151 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809154 2567 flags.go:64] FLAG: --port="10250" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809158 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809161 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0896cb2847c58bb88" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809164 2567 flags.go:64] FLAG: --qos-reserved="" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809167 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809170 2567 flags.go:64] FLAG: --register-node="true" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809173 2567 flags.go:64] FLAG: --register-with-taints="" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809177 2567 flags.go:64] FLAG: --registry-burst="10" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809180 2567 flags.go:64] FLAG: --registry-qps="5" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809182 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809185 2567 flags.go:64] FLAG: --reserved-memory="" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809189 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809192 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 04:09:13.816823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809195 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809198 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809201 2567 flags.go:64] FLAG: --runonce="false" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809204 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809207 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809210 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809213 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809216 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809219 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809223 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809225 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809228 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809231 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809234 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809236 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809239 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809244 2567 flags.go:64] FLAG: --system-cgroups="" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809247 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809252 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809255 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809258 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809261 2567 flags.go:64] FLAG: --tls-min-version="" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809264 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809267 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809270 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 04:09:13.817740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809273 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809276 2567 flags.go:64] FLAG: --v="2" Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809280 2567 flags.go:64] FLAG: --version="false" Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809284 2567 flags.go:64] FLAG: --vmodule="" Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809287 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.809291 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809420 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809432 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809436 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809438 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809441 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809444 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809447 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809450 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809453 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809456 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809459 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809461 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809464 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809466 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809469 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 04:09:13.818341 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809472 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809474 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809498 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809501 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809504 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809506 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809509 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809512 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809514 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809517 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809519 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809522 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809525 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809527 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809529 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809532 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809535 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809537 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809540 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 04:09:13.818907 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809548 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809551 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809554 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809556 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809559 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809561 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809564 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809566 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809569 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809571 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809574 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809576 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809579 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809583 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809586 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809591 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809593 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809596 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809599 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 04:09:13.819357 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809602 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809605 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809607 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809610 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809613 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809615 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809618 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809620 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809623 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809626 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809628 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809631 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809633 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809636 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809644 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809646 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809649 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809651 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809654 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809656 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 04:09:13.819910 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809659 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809663 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809666 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809668 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809671 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809675 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809678 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809681 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809685 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809687 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809690 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809692 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809697 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809699 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809702 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809704 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809707 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809709 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809712 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 04:09:13.820375 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809715 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809717 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809719 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809722 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809725 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809727 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.809730 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.810570 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.819083 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.819099 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.819119 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.819126 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819188 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819193 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819197 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 04:09:13.820861 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819200 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819204 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819208 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819212 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819216 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819219 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819222 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819225 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819227 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819230 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819233 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819237 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819241 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819244 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819247 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819250 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819253 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819255 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819258 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 04:09:13.821302 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819261 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819264 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819267 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819276 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819279 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819282 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819284 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819287 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819290 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819293 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819296 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819299 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819301 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819304 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819307 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819310 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819313 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819315 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819318 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819321 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 04:09:13.821781 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819323 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819326 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819329 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819333 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819337 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819340 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819343 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819346 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819348 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819351 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819354 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819357 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819360 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819364 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819366 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819369 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819372 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819375 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819377 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 04:09:13.822257 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819380 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819384 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819386 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819389 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819392 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819394 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819397 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819400 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819402 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819405 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819407 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819410 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819413 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819415 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819418 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819420 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819423 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819425 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819428 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819431 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 04:09:13.822717 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819433 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819436 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819439 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819441 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819444 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819447 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819449 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819452 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819455 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819457 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819460 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819463 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819466 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819469 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819471 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819474 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819476 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 04:09:13.823209 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819496 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.819518 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819648 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819654 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819657 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819660 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819663 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819666 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819669 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819672 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819675 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819678 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819681 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819684 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819688 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 04:09:13.823631 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819691 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819694 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819697 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819699 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819702 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819705 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819707 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819710 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819713 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819716 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819719 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819721 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819724 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819727 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819729 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819732 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819734 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819738 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819741 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819744 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 04:09:13.823995 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819747 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819750 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819752 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819754 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819757 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819760 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819762 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819765 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819768 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819770 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819773 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819776 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819782 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819784 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819787 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819790 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819792 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819795 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819797 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819800 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 04:09:13.824464 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819803 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819805 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819808 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819810 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819813 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819816 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819818 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819821 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819823 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819826 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819829 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819832 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819835 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819837 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819840 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819842 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819845 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819848 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819851 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 04:09:13.824953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819853 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819856 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819859 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819861 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819863 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819866 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819872 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819875 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819877 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819880 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819882 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819885 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819887 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819890 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819893 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819895 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819897 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819900 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819902 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 04:09:13.825409 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819906 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819910 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819912 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819916 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819919 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819923 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819926 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:13.819929 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.819934 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.820044 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.824052 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.825762 2567 server.go:1019] "Starting client certificate rotation" Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.825868 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 04:09:13.825953 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.825897 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 04:09:13.828573 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.828560 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 04:09:13.833034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.833015 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 04:09:13.844874 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.844855 2567 log.go:25] "Validated CRI v1 runtime API" Jul 21 04:09:13.847064 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.847019 2567 log.go:25] "Validated CRI v1 image API" Jul 21 04:09:13.847864 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.847850 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 04:09:13.851409 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.851386 2567 fs.go:135] Filesystem UUIDs: map[71adb991-5797-4cf5-9393-d8b578abbc50:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b0fdf54b-be6c-49f3-a3cf-e983cca4b4a0:/dev/nvme0n1p3] Jul 21 04:09:13.851469 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.851408 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 04:09:13.852821 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.852804 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 04:09:13.856572 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.856455 2567 manager.go:216] Machine: {Timestamp:2026-07-21 04:09:13.855114145 +0000 UTC m=+0.352302636 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099667 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f16b81a39a03f5f9443f1a1eb5098 SystemUUID:ec2f16b8-1a39-a03f-5f94-43f1a1eb5098 BootID:967047b5-4bd0-4c06-bbdb-6db39ea00679 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:54:b9:8f:89:9f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:54:b9:8f:89:9f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:0b:94:1f:27:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 04:09:13.856572 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.856570 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 04:09:13.857870 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.856841 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 04:09:13.860869 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.860689 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 04:09:13.861029 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.860877 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 04:09:13.861580 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.861569 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 04:09:13.861634 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.861585 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 04:09:13.861634 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.861595 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 04:09:13.862220 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.862209 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 04:09:13.863448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.863435 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 04:09:13.866338 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.866324 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 21 04:09:13.866470 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.866459 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 04:09:13.866546 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.866510 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 21 04:09:13.866546 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.866524 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 04:09:13.866546 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.866543 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 04:09:13.866678 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.866556 2567 kubelet.go:398] "Adding apiserver pod source" Jul 21 04:09:13.866678 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.866568 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 04:09:13.866928 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.866915 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 04:09:13.866992 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.866936 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 04:09:13.869051 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.869035 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 04:09:13.871068 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.871054 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 04:09:13.871112 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.871084 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 04:09:13.872103 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872091 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 04:09:13.872158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872107 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 04:09:13.872158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872113 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 04:09:13.872158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872119 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 04:09:13.872158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872127 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 04:09:13.872158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872133 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 04:09:13.872158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872141 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 04:09:13.872158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872146 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 04:09:13.872158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872152 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 04:09:13.872158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872157 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 04:09:13.872425 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872165 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 04:09:13.872425 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.872176 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 04:09:13.873726 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.873715 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 04:09:13.873726 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.873726 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 04:09:13.876146 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.876126 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 04:09:13.876757 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:13.876736 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 04:09:13.876817 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:13.876789 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 04:09:13.876854 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.876836 2567 server.go:1268] "Started kubelet" Jul 21 04:09:13.876949 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.876921 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 04:09:13.877034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.876935 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 04:09:13.877034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.876999 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 04:09:13.877577 ip-10-0-137-226 systemd[1]: Started Kubernetes Kubelet. Jul 21 04:09:13.879569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.878384 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 21 04:09:13.879569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.878608 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 04:09:13.882466 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:13.882444 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 04:09:13.885871 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:13.884613 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-226.ec2.internal.18c43291bd84ecfe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-226.ec2.internal,UID:ip-10-0-137-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-226.ec2.internal,},FirstTimestamp:2026-07-21 04:09:13.876811006 +0000 UTC m=+0.373999497,LastTimestamp:2026-07-21 04:09:13.876811006 +0000 UTC m=+0.373999497,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-226.ec2.internal,}" Jul 21 04:09:13.885871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.885842 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 04:09:13.885995 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.885851 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 04:09:13.885995 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:13.885962 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:13.886540 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886525 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 04:09:13.886672 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886654 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 04:09:13.886761 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886700 2567 factory.go:55] Registering systemd factory Jul 21 04:09:13.886761 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886716 2567 factory.go:223] Registration of the systemd container factory successfully Jul 21 04:09:13.886761 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886525 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 04:09:13.886895 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886798 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 21 04:09:13.886895 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886806 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 21 04:09:13.886986 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886933 2567 factory.go:153] Registering CRI-O factory Jul 21 04:09:13.886986 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886947 2567 factory.go:223] Registration of the crio container factory successfully Jul 21 04:09:13.887073 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.886996 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 04:09:13.887073 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.887017 2567 factory.go:103] Registering Raw factory Jul 21 04:09:13.887073 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.887032 2567 manager.go:1196] Started watching for new ooms in manager Jul 21 04:09:13.887524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.887507 2567 manager.go:349] Starting recovery of all containers Jul 21 04:09:13.895599 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:13.895289 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 04:09:13.895709 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:13.895658 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 04:09:13.899429 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.899404 2567 manager.go:354] Recovery completed Jul 21 04:09:13.902560 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.902538 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zsxd2" Jul 21 04:09:13.904033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.904022 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 04:09:13.905897 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.905876 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory" Jul 21 04:09:13.905974 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.905925 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 04:09:13.905974 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.905942 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID" Jul 21 04:09:13.906460 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.906446 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 04:09:13.906460 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.906461 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 04:09:13.906569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.906468 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 04:09:13.906569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.906475 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 21 04:09:13.907219 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:13.907127 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-226.ec2.internal.18c43291bf40bd79 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-226.ec2.internal,UID:ip-10-0-137-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-226.ec2.internal,},FirstTimestamp:2026-07-21 04:09:13.905896825 +0000 UTC m=+0.403085324,LastTimestamp:2026-07-21 04:09:13.905896825 +0000 UTC m=+0.403085324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-226.ec2.internal,}" Jul 21 04:09:13.907315 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.907197 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory" Jul 21 04:09:13.907315 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.907308 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 04:09:13.907417 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.907319 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID" Jul 21 04:09:13.910158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:13.910144 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zsxd2" Jul 21 04:09:13.986982 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:13.986938 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:14.058249 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.058231 2567 policy_none.go:49] "None policy: Start" Jul 21 04:09:14.058311 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.058253 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 04:09:14.058311 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.058265 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 04:09:14.059451 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.059440 2567 policy_none.go:47] "Start" Jul 21 04:09:14.088148 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.088006 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:14.090039 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.090026 2567 manager.go:337] "Starting Device Plugin manager" Jul 21 04:09:14.090136 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.090065 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 04:09:14.090136 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.090074 2567 server.go:87] "Starting device plugin registration server" Jul 21 04:09:14.090330 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.090319 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 04:09:14.090381 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.090334 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 04:09:14.090459 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.090444 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 04:09:14.090561 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.090542 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 04:09:14.090561 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.090551 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 04:09:14.091086 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.091063 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 04:09:14.091168 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.091122 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:14.102213 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.102167 2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-226.ec2.internal\" not found" node="ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.160773 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.160741 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 04:09:14.161848 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.161833 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 04:09:14.161921 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.161869 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 04:09:14.161921 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.161888 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 04:09:14.162028 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.161919 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 04:09:14.164385 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.164368 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 04:09:14.190715 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.190700 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 04:09:14.191552 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.191536 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory" Jul 21 04:09:14.191621 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.191566 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 04:09:14.191621 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.191579 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID" Jul 21 04:09:14.191621 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.191599 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.198931 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.198918 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.198972 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.198938 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-226.ec2.internal\": node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:14.223913 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.223892 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:14.263815 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.263790 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal"] Jul 21 04:09:14.263891 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.263861 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 04:09:14.264704 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.264687 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory" Jul 21 04:09:14.264794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.264720 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 04:09:14.264794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.264734 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID" Jul 21 04:09:14.266047 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.266032 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 04:09:14.266151 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.266138 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.266198 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.266164 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 04:09:14.266700 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.266684 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory" Jul 21 04:09:14.266787 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.266703 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory" Jul 21 04:09:14.266787 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.266713 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 04:09:14.266787 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.266730 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 04:09:14.266787 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.266733 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID" Jul 21 04:09:14.266787 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.266742 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID" Jul 21 04:09:14.267865 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.267847 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.267948 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.267873 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 04:09:14.268506 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.268476 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientMemory" Jul 21 04:09:14.268605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.268525 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 04:09:14.268605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.268537 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeHasSufficientPID" Jul 21 04:09:14.287586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.287565 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1941da6dbd8a4efb4a368091e4943a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"f1941da6dbd8a4efb4a368091e4943a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.292203 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.292186 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-226.ec2.internal\" not found" node="ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.296550 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.296534 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-226.ec2.internal\" not found" node="ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.324027 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.324006 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:14.387950 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.387877 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1941da6dbd8a4efb4a368091e4943a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"f1941da6dbd8a4efb4a368091e4943a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.387950 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.387911 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1941da6dbd8a4efb4a368091e4943a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"f1941da6dbd8a4efb4a368091e4943a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.387950 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.387928 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e07ae959d76e4e217922a691b6e844d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-226.ec2.internal\" (UID: \"3e07ae959d76e4e217922a691b6e844d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.388125 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.387962 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f1941da6dbd8a4efb4a368091e4943a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"f1941da6dbd8a4efb4a368091e4943a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.425038 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.425011 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:14.488399 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.488364 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1941da6dbd8a4efb4a368091e4943a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"f1941da6dbd8a4efb4a368091e4943a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.488516 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.488404 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e07ae959d76e4e217922a691b6e844d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-226.ec2.internal\" (UID: \"3e07ae959d76e4e217922a691b6e844d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.488516 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.488429 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e07ae959d76e4e217922a691b6e844d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-226.ec2.internal\" (UID: \"3e07ae959d76e4e217922a691b6e844d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.488516 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.488463 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1941da6dbd8a4efb4a368091e4943a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal\" (UID: \"f1941da6dbd8a4efb4a368091e4943a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.525510 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.525473 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:14.594981 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.594949 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.599823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:14.599807 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" Jul 21 04:09:14.626434 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:14.626410 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:15.824841 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:15.824815 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:15.922236 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:15.922205 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 04:09:15.922702 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:15.922386 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 04:09:15.925382 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:15.925362 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:16.012633 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.012599 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 04:04:15 +0000 UTC" deadline="2027-12-19 06:37:45.133946625 +0000 UTC" Jul 21 04:09:16.012633 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.012627 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12386h28m29.121322446s" Jul 21 04:09:16.025755 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:16.025736 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:16.126251 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:16.126224 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:16.205429 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:16.205396 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e07ae959d76e4e217922a691b6e844d.slice/crio-070ec7f850d0b9d5ec4da170ef78b8adef81293621f41e4f0dbc60b0f4a161ce WatchSource:0}: Error finding container 070ec7f850d0b9d5ec4da170ef78b8adef81293621f41e4f0dbc60b0f4a161ce: Status 404 returned error can't find the container with id 070ec7f850d0b9d5ec4da170ef78b8adef81293621f41e4f0dbc60b0f4a161ce Jul 21 04:09:16.205670 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:16.205657 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1941da6dbd8a4efb4a368091e4943a9.slice/crio-db6fa7e75b637ba554e9a6718b8d33cea53fcebe0b6e2860e9e3885e2b2e0597 WatchSource:0}: Error finding container db6fa7e75b637ba554e9a6718b8d33cea53fcebe0b6e2860e9e3885e2b2e0597: Status 404 returned error can't find the container with id db6fa7e75b637ba554e9a6718b8d33cea53fcebe0b6e2860e9e3885e2b2e0597 Jul 21 04:09:16.210606 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.210592 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 04:09:16.227177 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:16.227157 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:16.263492 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.263447 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" event={"ID":"3e07ae959d76e4e217922a691b6e844d","Type":"ContainerStarted","Data":"070ec7f850d0b9d5ec4da170ef78b8adef81293621f41e4f0dbc60b0f4a161ce"} Jul 21 04:09:16.264529 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.264507 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" event={"ID":"f1941da6dbd8a4efb4a368091e4943a9","Type":"ContainerStarted","Data":"db6fa7e75b637ba554e9a6718b8d33cea53fcebe0b6e2860e9e3885e2b2e0597"} Jul 21 04:09:16.327973 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:16.327935 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:16.428702 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:16.428625 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:16.436155 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.436130 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 04:09:16.488779 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.488751 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 04:09:16.529749 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:16.529707 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-226.ec2.internal\" not found" Jul 21 04:09:16.575428 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.575405 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 04:09:16.584262 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.584243 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" Jul 21 04:09:16.592017 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.592001 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 04:09:16.593011 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.592998 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" Jul 21 04:09:16.604509 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.604494 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 04:09:16.872523 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.872492 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 04:09:16.965061 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.965026 2567 apiserver.go:52] "Watching apiserver" Jul 21 04:09:16.972996 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.972971 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 04:09:16.975103 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.975075 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-2jgjj","openshift-dns/node-resolver-bkscj","openshift-multus/multus-897jc","openshift-multus/network-metrics-daemon-8w86n","openshift-network-operator/iptables-alerter-mndqf","openshift-ovn-kubernetes/ovnkube-node-m2ssv","kube-system/global-pull-secret-syncer-wn4dg","kube-system/konnectivity-agent-j676g","kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df","openshift-image-registry/node-ca-snqhr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal","openshift-multus/multus-additional-cni-plugins-dtmdb","openshift-network-diagnostics/network-check-target-ft8hx"] Jul 21 04:09:16.976625 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.976603 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:16.976734 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:16.976689 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:16.978180 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.978160 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:16.979770 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.979747 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-897jc" Jul 21 04:09:16.981260 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.981211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:16.981368 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:16.981287 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:16.981815 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.981531 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 04:09:16.981815 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.981576 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 04:09:16.981815 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.981811 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 04:09:16.981993 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.981822 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nnsbl\"" Jul 21 04:09:16.981993 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.981929 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 04:09:16.983422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.982502 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 04:09:16.983422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.982547 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:16.983422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.982840 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4fpgg\"" Jul 21 04:09:16.983422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.983050 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 04:09:16.984251 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.984153 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.985397 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.985073 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r59jz\"" Jul 21 04:09:16.985397 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.985081 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 04:09:16.985397 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.985209 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 04:09:16.985594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.985432 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 04:09:16.985831 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.985698 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 04:09:16.986716 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.986032 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:16.986716 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:16.986093 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:16.986716 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.986479 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 04:09:16.986716 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.986559 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 04:09:16.987249 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.987161 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 04:09:16.987499 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.987411 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:16.990199 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.990179 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 04:09:16.990427 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.990410 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 04:09:16.994520 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.991962 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vk4md\"" Jul 21 04:09:16.994520 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.992149 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 04:09:16.994520 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.993005 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 04:09:16.994520 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.993288 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7gfs7\"" Jul 21 04:09:16.994520 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.993520 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 04:09:16.994780 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.994645 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995145 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-kubelet-config\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-var-lib-cni-bin\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-os-release\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995247 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-multus-socket-dir-parent\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995278 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-cni-netd\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995308 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-multus-conf-dir\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995362 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af409d50-2522-42a5-bada-8126ed1944c3-multus-daemon-config\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995393 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-run-openvswitch\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995444 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995473 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-cnibin\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995503 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-run-netns\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995534 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hgjr\" (UniqueName: \"kubernetes.io/projected/3f65e21d-704f-40f0-b476-f76d71d79fff-kube-api-access-2hgjr\") pod \"iptables-alerter-mndqf\" (UID: \"3f65e21d-704f-40f0-b476-f76d71d79fff\") " pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995578 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-dbus\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995604 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-os-release\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995639 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-run-k8s-cni-cncf-io\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995777 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-var-lib-cni-multus\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.995867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995829 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-etc-kubernetes\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-run-netns\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995903 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-run-systemd\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995935 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995966 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-hostroot\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.995997 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-run-ovn\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-multus-cni-dir\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996052 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2rh\" (UniqueName: \"kubernetes.io/projected/fabd6623-e601-4828-8b83-673db7105de6-kube-api-access-lr2rh\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996082 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-var-lib-openvswitch\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e5da763-6747-4693-af09-9370b50f6e56-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996147 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwl4\" (UniqueName: \"kubernetes.io/projected/3e5da763-6747-4693-af09-9370b50f6e56-kube-api-access-tpwl4\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996181 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-cnibin\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e5da763-6747-4693-af09-9370b50f6e56-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996239 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af409d50-2522-42a5-bada-8126ed1944c3-cni-binary-copy\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996270 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996299 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f65e21d-704f-40f0-b476-f76d71d79fff-host-slash\") pod \"iptables-alerter-mndqf\" (UID: \"3f65e21d-704f-40f0-b476-f76d71d79fff\") " pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:16.996753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996329 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-kubelet\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996371 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-etc-openvswitch\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996401 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-node-log\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996432 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e5da763-6747-4693-af09-9370b50f6e56-cni-binary-copy\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996463 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/241293ba-5312-43da-b1d6-7ba42c71cfbd-env-overrides\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-system-cni-dir\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996531 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-449ls\" (UniqueName: \"kubernetes.io/projected/af409d50-2522-42a5-bada-8126ed1944c3-kube-api-access-449ls\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f65e21d-704f-40f0-b476-f76d71d79fff-iptables-alerter-script\") pod \"iptables-alerter-mndqf\" (UID: \"3f65e21d-704f-40f0-b476-f76d71d79fff\") " pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996601 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-systemd-units\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996633 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-slash\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996666 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-run-multus-certs\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996691 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-cni-bin\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/241293ba-5312-43da-b1d6-7ba42c71cfbd-ovnkube-script-lib\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996754 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-system-cni-dir\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996793 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-log-socket\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996823 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/241293ba-5312-43da-b1d6-7ba42c71cfbd-ovn-node-metrics-cert\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.997524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996863 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:16.998279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996888 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-var-lib-kubelet\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:16.998279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996918 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/241293ba-5312-43da-b1d6-7ba42c71cfbd-ovnkube-config\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.998279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996949 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn97x\" (UniqueName: \"kubernetes.io/projected/241293ba-5312-43da-b1d6-7ba42c71cfbd-kube-api-access-zn97x\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.998279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.996980 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:16.998279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.997011 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:16.998279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.997293 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:16.998279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.997451 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 04:09:16.998279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.997480 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:16.998279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.997509 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wb4gj\"" Jul 21 04:09:16.998708 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.998424 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 04:09:17.001201 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:16.999859 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:17.001201 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.000604 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 04:09:17.001201 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.000828 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2j8hp\"" Jul 21 04:09:17.001201 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.000975 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 04:09:17.002961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.001666 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w5dpb\"" Jul 21 04:09:17.002961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.001889 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 04:09:17.002961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.002068 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 04:09:17.002961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.002125 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lmg9s\"" Jul 21 04:09:17.002961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.002246 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 04:09:17.002961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.002312 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 04:09:17.002961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.002483 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 04:09:17.002961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.002495 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 04:09:17.002961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.002764 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 04:09:17.021096 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.021073 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6mzf2" Jul 21 04:09:17.029585 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.029564 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6mzf2" Jul 21 04:09:17.085894 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.085868 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 04:09:17.097969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.097929 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hgjr\" (UniqueName: \"kubernetes.io/projected/3f65e21d-704f-40f0-b476-f76d71d79fff-kube-api-access-2hgjr\") pod \"iptables-alerter-mndqf\" (UID: \"3f65e21d-704f-40f0-b476-f76d71d79fff\") " pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:17.097969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.097968 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-dbus\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:17.098139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.097988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-os-release\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.098139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098003 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-run-k8s-cni-cncf-io\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.098139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098021 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-var-lib-cni-multus\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.098139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098036 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-etc-kubernetes\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.098139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-run-netns\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.098139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098067 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-run-systemd\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.098139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098098 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.098139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ab7d0e5-db4f-4988-80a9-4edbd26c174c-host\") pod \"node-ca-snqhr\" (UID: \"5ab7d0e5-db4f-4988-80a9-4edbd26c174c\") " pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.098139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-hostroot\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvxh\" (UniqueName: \"kubernetes.io/projected/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-kube-api-access-8kvxh\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a894722a-d3ee-4318-9673-6a1962e067e5-tmp\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098195 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-run-ovn\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098209 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-registration-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098224 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-var-lib-kubelet\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-multus-cni-dir\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098255 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2rh\" (UniqueName: \"kubernetes.io/projected/fabd6623-e601-4828-8b83-673db7105de6-kube-api-access-lr2rh\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-var-lib-openvswitch\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e5da763-6747-4693-af09-9370b50f6e56-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwl4\" (UniqueName: \"kubernetes.io/projected/3e5da763-6747-4693-af09-9370b50f6e56-kube-api-access-tpwl4\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098318 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-cnibin\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098368 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e5da763-6747-4693-af09-9370b50f6e56-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098384 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af409d50-2522-42a5-bada-8126ed1944c3-cni-binary-copy\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098403 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:17.098586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f65e21d-704f-40f0-b476-f76d71d79fff-host-slash\") pod \"iptables-alerter-mndqf\" (UID: \"3f65e21d-704f-40f0-b476-f76d71d79fff\") " pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-kubelet\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098489 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-etc-openvswitch\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-node-log\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098518 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e5da763-6747-4693-af09-9370b50f6e56-cni-binary-copy\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/241293ba-5312-43da-b1d6-7ba42c71cfbd-env-overrides\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098552 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-system-cni-dir\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098576 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-449ls\" (UniqueName: \"kubernetes.io/projected/af409d50-2522-42a5-bada-8126ed1944c3-kube-api-access-449ls\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098609 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f65e21d-704f-40f0-b476-f76d71d79fff-iptables-alerter-script\") pod \"iptables-alerter-mndqf\" (UID: \"3f65e21d-704f-40f0-b476-f76d71d79fff\") " pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098628 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-systemd-units\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-slash\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098683 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/406658cc-5989-4a5e-8ef8-a27780de41a5-hosts-file\") pod \"node-resolver-bkscj\" (UID: \"406658cc-5989-4a5e-8ef8-a27780de41a5\") " pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-run-multus-certs\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098730 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-cni-bin\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098751 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/241293ba-5312-43da-b1d6-7ba42c71cfbd-ovnkube-script-lib\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-system-cni-dir\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ab7d0e5-db4f-4988-80a9-4edbd26c174c-serviceca\") pod \"node-ca-snqhr\" (UID: \"5ab7d0e5-db4f-4988-80a9-4edbd26c174c\") " pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.099290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098816 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-host\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098831 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a894722a-d3ee-4318-9673-6a1962e067e5-etc-tuned\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098852 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-log-socket\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098879 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/241293ba-5312-43da-b1d6-7ba42c71cfbd-ovn-node-metrics-cert\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098914 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098940 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-socket-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098965 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.098988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-run\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-sys\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099034 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-var-lib-kubelet\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099060 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/241293ba-5312-43da-b1d6-7ba42c71cfbd-ovnkube-config\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099088 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn97x\" (UniqueName: \"kubernetes.io/projected/241293ba-5312-43da-b1d6-7ba42c71cfbd-kube-api-access-zn97x\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099139 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-sysconfig\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.099153 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099168 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099192 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406658cc-5989-4a5e-8ef8-a27780de41a5-tmp-dir\") pod \"node-resolver-bkscj\" (UID: \"406658cc-5989-4a5e-8ef8-a27780de41a5\") " pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.100058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099217 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6254d1d7-216d-4417-9fcb-6f71dc38129b-agent-certs\") pod \"konnectivity-agent-j676g\" (UID: \"6254d1d7-216d-4417-9fcb-6f71dc38129b\") " pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.099249 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs podName:fabd6623-e601-4828-8b83-673db7105de6 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:17.599215472 +0000 UTC m=+2.998548227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs") pod "network-metrics-daemon-8w86n" (UID: "fabd6623-e601-4828-8b83-673db7105de6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099276 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f65e21d-704f-40f0-b476-f76d71d79fff-host-slash\") pod \"iptables-alerter-mndqf\" (UID: \"3f65e21d-704f-40f0-b476-f76d71d79fff\") " pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099327 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-kubelet\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099400 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-etc-openvswitch\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099441 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-node-log\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099673 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-os-release\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099707 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-dbus\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099726 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-run-k8s-cni-cncf-io\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099787 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-var-lib-cni-multus\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099841 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-multus-cni-dir\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099885 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-etc-kubernetes\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-run-netns\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099980 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-run-systemd\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099991 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3e5da763-6747-4693-af09-9370b50f6e56-cni-binary-copy\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100031 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100089 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-hostroot\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.100871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6254d1d7-216d-4417-9fcb-6f71dc38129b-konnectivity-ca\") pod \"konnectivity-agent-j676g\" (UID: \"6254d1d7-216d-4417-9fcb-6f71dc38129b\") " pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100160 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-run-ovn\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-kubernetes\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtshc\" (UniqueName: \"kubernetes.io/projected/406658cc-5989-4a5e-8ef8-a27780de41a5-kube-api-access-jtshc\") pod \"node-resolver-bkscj\" (UID: \"406658cc-5989-4a5e-8ef8-a27780de41a5\") " pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100261 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-lib-modules\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.099906 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-cnibin\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-kubelet-config\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100331 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-kubelet-config\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100360 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-var-lib-cni-bin\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100393 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-sys-fs\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100420 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5skqf\" (UniqueName: \"kubernetes.io/projected/5ab7d0e5-db4f-4988-80a9-4edbd26c174c-kube-api-access-5skqf\") pod \"node-ca-snqhr\" (UID: \"5ab7d0e5-db4f-4988-80a9-4edbd26c174c\") " pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100449 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-os-release\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-multus-socket-dir-parent\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af409d50-2522-42a5-bada-8126ed1944c3-cni-binary-copy\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100502 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-device-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100502 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/241293ba-5312-43da-b1d6-7ba42c71cfbd-env-overrides\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100607 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3e5da763-6747-4693-af09-9370b50f6e56-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.101690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3e5da763-6747-4693-af09-9370b50f6e56-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100929 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-slash\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100935 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-system-cni-dir\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100941 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-systemd-units\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100970 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-modprobe-d\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101018 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-cni-bin\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101051 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-cni-netd\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101057 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-var-lib-cni-bin\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101106 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-system-cni-dir\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-run-ovn-kubernetes\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.100986 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-var-lib-kubelet\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101192 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-run-multus-certs\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101244 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-multus-socket-dir-parent\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3e5da763-6747-4693-af09-9370b50f6e56-os-release\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101333 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-var-lib-openvswitch\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101533 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/241293ba-5312-43da-b1d6-7ba42c71cfbd-ovnkube-script-lib\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101532 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-log-socket\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.102433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101021 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-host-cni-netd\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101593 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-multus-conf-dir\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101625 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af409d50-2522-42a5-bada-8126ed1944c3-multus-daemon-config\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101619 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f65e21d-704f-40f0-b476-f76d71d79fff-iptables-alerter-script\") pod \"iptables-alerter-mndqf\" (UID: \"3f65e21d-704f-40f0-b476-f76d71d79fff\") " pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101637 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/241293ba-5312-43da-b1d6-7ba42c71cfbd-ovnkube-config\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101645 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-run-openvswitch\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101661 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-multus-conf-dir\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-sysctl-d\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101704 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/241293ba-5312-43da-b1d6-7ba42c71cfbd-run-openvswitch\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101707 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-systemd\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101741 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101771 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-cnibin\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.101805 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101819 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-run-netns\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101850 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-cnibin\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101882 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-sysctl-conf\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101917 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af409d50-2522-42a5-bada-8126ed1944c3-host-run-netns\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.103154 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.101923 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret podName:c21ab5db-d116-4b5f-8f61-df4fe3a0ef89 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:17.601903298 +0000 UTC m=+3.001236049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret") pod "global-pull-secret-syncer-wn4dg" (UID: "c21ab5db-d116-4b5f-8f61-df4fe3a0ef89") : object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:17.103940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.101961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7qw\" (UniqueName: \"kubernetes.io/projected/a894722a-d3ee-4318-9673-6a1962e067e5-kube-api-access-wl7qw\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.103940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.102043 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/af409d50-2522-42a5-bada-8126ed1944c3-multus-daemon-config\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.105584 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.105555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/241293ba-5312-43da-b1d6-7ba42c71cfbd-ovn-node-metrics-cert\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.107778 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.107761 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 04:09:17.107871 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.107783 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 04:09:17.107871 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.107795 2567 projected.go:196] Error preparing data for projected volume kube-api-access-mvwfz for pod openshift-network-diagnostics/network-check-target-ft8hx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:17.107871 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.107849 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz podName:3fff3e92-f836-43fd-8c86-f79e0712cdd1 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:17.607833042 +0000 UTC m=+3.007165778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mvwfz" (UniqueName: "kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz") pod "network-check-target-ft8hx" (UID: "3fff3e92-f836-43fd-8c86-f79e0712cdd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:17.109400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.109370 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2rh\" (UniqueName: \"kubernetes.io/projected/fabd6623-e601-4828-8b83-673db7105de6-kube-api-access-lr2rh\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:17.109807 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.109782 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hgjr\" (UniqueName: \"kubernetes.io/projected/3f65e21d-704f-40f0-b476-f76d71d79fff-kube-api-access-2hgjr\") pod \"iptables-alerter-mndqf\" (UID: \"3f65e21d-704f-40f0-b476-f76d71d79fff\") " pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:17.110995 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.110956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwl4\" (UniqueName: \"kubernetes.io/projected/3e5da763-6747-4693-af09-9370b50f6e56-kube-api-access-tpwl4\") pod \"multus-additional-cni-plugins-dtmdb\" (UID: \"3e5da763-6747-4693-af09-9370b50f6e56\") " pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.111151 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.111131 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn97x\" (UniqueName: \"kubernetes.io/projected/241293ba-5312-43da-b1d6-7ba42c71cfbd-kube-api-access-zn97x\") pod \"ovnkube-node-m2ssv\" (UID: \"241293ba-5312-43da-b1d6-7ba42c71cfbd\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.111572 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.111548 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-449ls\" (UniqueName: \"kubernetes.io/projected/af409d50-2522-42a5-bada-8126ed1944c3-kube-api-access-449ls\") pod \"multus-897jc\" (UID: \"af409d50-2522-42a5-bada-8126ed1944c3\") " pod="openshift-multus/multus-897jc" Jul 21 04:09:17.202449 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a894722a-d3ee-4318-9673-6a1962e067e5-etc-tuned\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.202449 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202406 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-socket-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.202449 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.202449 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202448 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-run\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202466 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-sys\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-sys\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202570 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-run\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-sysconfig\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202652 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-socket-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202661 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406658cc-5989-4a5e-8ef8-a27780de41a5-tmp-dir\") pod \"node-resolver-bkscj\" (UID: \"406658cc-5989-4a5e-8ef8-a27780de41a5\") " pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6254d1d7-216d-4417-9fcb-6f71dc38129b-agent-certs\") pod \"konnectivity-agent-j676g\" (UID: \"6254d1d7-216d-4417-9fcb-6f71dc38129b\") " pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-sysconfig\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6254d1d7-216d-4417-9fcb-6f71dc38129b-konnectivity-ca\") pod \"konnectivity-agent-j676g\" (UID: \"6254d1d7-216d-4417-9fcb-6f71dc38129b\") " pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:17.202762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202734 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-kubernetes\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202785 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-kubernetes\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtshc\" (UniqueName: \"kubernetes.io/projected/406658cc-5989-4a5e-8ef8-a27780de41a5-kube-api-access-jtshc\") pod \"node-resolver-bkscj\" (UID: \"406658cc-5989-4a5e-8ef8-a27780de41a5\") " pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202915 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-lib-modules\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202947 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/406658cc-5989-4a5e-8ef8-a27780de41a5-tmp-dir\") pod \"node-resolver-bkscj\" (UID: \"406658cc-5989-4a5e-8ef8-a27780de41a5\") " pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-sys-fs\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.202978 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5skqf\" (UniqueName: \"kubernetes.io/projected/5ab7d0e5-db4f-4988-80a9-4edbd26c174c-kube-api-access-5skqf\") pod \"node-ca-snqhr\" (UID: \"5ab7d0e5-db4f-4988-80a9-4edbd26c174c\") " pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203002 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-device-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-modprobe-d\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-sysctl-d\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-lib-modules\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-systemd\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203123 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-sysctl-conf\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203147 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7qw\" (UniqueName: \"kubernetes.io/projected/a894722a-d3ee-4318-9673-6a1962e067e5-kube-api-access-wl7qw\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203179 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ab7d0e5-db4f-4988-80a9-4edbd26c174c-host\") pod \"node-ca-snqhr\" (UID: \"5ab7d0e5-db4f-4988-80a9-4edbd26c174c\") " pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203186 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-device-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203212 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvxh\" (UniqueName: \"kubernetes.io/projected/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-kube-api-access-8kvxh\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.203289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a894722a-d3ee-4318-9673-6a1962e067e5-tmp\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203272 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-registration-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203293 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-var-lib-kubelet\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203321 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203327 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-sysctl-conf\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203352 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-sysctl-d\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203355 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-sys-fs\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203402 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/406658cc-5989-4a5e-8ef8-a27780de41a5-hosts-file\") pod \"node-resolver-bkscj\" (UID: \"406658cc-5989-4a5e-8ef8-a27780de41a5\") " pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203416 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-registration-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203406 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6254d1d7-216d-4417-9fcb-6f71dc38129b-konnectivity-ca\") pod \"konnectivity-agent-j676g\" (UID: \"6254d1d7-216d-4417-9fcb-6f71dc38129b\") " pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ab7d0e5-db4f-4988-80a9-4edbd26c174c-serviceca\") pod \"node-ca-snqhr\" (UID: \"5ab7d0e5-db4f-4988-80a9-4edbd26c174c\") " pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203458 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-systemd\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203473 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ab7d0e5-db4f-4988-80a9-4edbd26c174c-host\") pod \"node-ca-snqhr\" (UID: \"5ab7d0e5-db4f-4988-80a9-4edbd26c174c\") " pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203488 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-var-lib-kubelet\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203424 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203498 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-host\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203573 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/406658cc-5989-4a5e-8ef8-a27780de41a5-hosts-file\") pod \"node-resolver-bkscj\" (UID: \"406658cc-5989-4a5e-8ef8-a27780de41a5\") " pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203589 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-host\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.204119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203655 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a894722a-d3ee-4318-9673-6a1962e067e5-etc-modprobe-d\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.205064 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.203906 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ab7d0e5-db4f-4988-80a9-4edbd26c174c-serviceca\") pod \"node-ca-snqhr\" (UID: \"5ab7d0e5-db4f-4988-80a9-4edbd26c174c\") " pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.205898 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.205851 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a894722a-d3ee-4318-9673-6a1962e067e5-etc-tuned\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.206147 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.206122 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a894722a-d3ee-4318-9673-6a1962e067e5-tmp\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.206930 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.206903 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6254d1d7-216d-4417-9fcb-6f71dc38129b-agent-certs\") pod \"konnectivity-agent-j676g\" (UID: \"6254d1d7-216d-4417-9fcb-6f71dc38129b\") " pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:17.212165 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.212121 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvxh\" (UniqueName: \"kubernetes.io/projected/3d359fcb-2a3e-4859-a07c-1a8ceb9d904c-kube-api-access-8kvxh\") pod \"aws-ebs-csi-driver-node-qw7df\" (UID: \"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.212284 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.212257 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5skqf\" (UniqueName: \"kubernetes.io/projected/5ab7d0e5-db4f-4988-80a9-4edbd26c174c-kube-api-access-5skqf\") pod \"node-ca-snqhr\" (UID: \"5ab7d0e5-db4f-4988-80a9-4edbd26c174c\") " pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.212524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.212500 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7qw\" (UniqueName: \"kubernetes.io/projected/a894722a-d3ee-4318-9673-6a1962e067e5-kube-api-access-wl7qw\") pod \"tuned-2jgjj\" (UID: \"a894722a-d3ee-4318-9673-6a1962e067e5\") " pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.212947 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.212923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtshc\" (UniqueName: \"kubernetes.io/projected/406658cc-5989-4a5e-8ef8-a27780de41a5-kube-api-access-jtshc\") pod \"node-resolver-bkscj\" (UID: \"406658cc-5989-4a5e-8ef8-a27780de41a5\") " pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.302208 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.302176 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" Jul 21 04:09:17.311310 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.311285 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-897jc" Jul 21 04:09:17.320375 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.320354 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mndqf" Jul 21 04:09:17.326240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.326221 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:17.335215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.335197 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" Jul 21 04:09:17.342300 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.342276 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bkscj" Jul 21 04:09:17.349954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.349934 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" Jul 21 04:09:17.357489 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.357468 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-snqhr" Jul 21 04:09:17.364310 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.364232 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:17.606580 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.606549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:17.606738 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.606603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:17.606738 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.606695 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:17.606738 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.606698 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:17.606858 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.606756 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret podName:c21ab5db-d116-4b5f-8f61-df4fe3a0ef89 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:18.606742547 +0000 UTC m=+4.006075283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret") pod "global-pull-secret-syncer-wn4dg" (UID: "c21ab5db-d116-4b5f-8f61-df4fe3a0ef89") : object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:17.606858 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.606769 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs podName:fabd6623-e601-4828-8b83-673db7105de6 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:18.606763691 +0000 UTC m=+4.006096426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs") pod "network-metrics-daemon-8w86n" (UID: "fabd6623-e601-4828-8b83-673db7105de6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:17.707603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:17.707568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:17.707768 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.707742 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 04:09:17.707817 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.707769 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 04:09:17.707817 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.707783 2567 projected.go:196] Error preparing data for projected volume kube-api-access-mvwfz for pod openshift-network-diagnostics/network-check-target-ft8hx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:17.707902 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:17.707840 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz podName:3fff3e92-f836-43fd-8c86-f79e0712cdd1 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:18.70782315 +0000 UTC m=+4.107155887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvwfz" (UniqueName: "kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz") pod "network-check-target-ft8hx" (UID: "3fff3e92-f836-43fd-8c86-f79e0712cdd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:18.031386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.031308 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 04:04:17 +0000 UTC" deadline="2028-05-05 07:13:19.191054714 +0000 UTC" Jul 21 04:09:18.031386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.031382 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15699h4m1.159677082s" Jul 21 04:09:18.181700 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:18.181415 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f65e21d_704f_40f0_b476_f76d71d79fff.slice/crio-4b724bc3d7e4e80748a3899ca1e5f75a18dcce99e086e4c2b99cb8cbc0da9b5e WatchSource:0}: Error finding container 4b724bc3d7e4e80748a3899ca1e5f75a18dcce99e086e4c2b99cb8cbc0da9b5e: Status 404 returned error can't find the container with id 4b724bc3d7e4e80748a3899ca1e5f75a18dcce99e086e4c2b99cb8cbc0da9b5e Jul 21 04:09:18.184593 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:18.184362 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d359fcb_2a3e_4859_a07c_1a8ceb9d904c.slice/crio-24b40d3e105221b79f0b3e8884e7a0e6681863bc093d5acc10110ffafd00578b WatchSource:0}: Error finding container 24b40d3e105221b79f0b3e8884e7a0e6681863bc093d5acc10110ffafd00578b: Status 404 returned error can't find the container with id 24b40d3e105221b79f0b3e8884e7a0e6681863bc093d5acc10110ffafd00578b Jul 21 04:09:18.186713 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:18.186688 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241293ba_5312_43da_b1d6_7ba42c71cfbd.slice/crio-45557716ce669ff9768e607da8e307e477329bdc8c91d95cb3a568b567ac44e1 WatchSource:0}: Error finding container 45557716ce669ff9768e607da8e307e477329bdc8c91d95cb3a568b567ac44e1: Status 404 returned error can't find the container with id 45557716ce669ff9768e607da8e307e477329bdc8c91d95cb3a568b567ac44e1 Jul 21 04:09:18.187688 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:18.187662 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf409d50_2522_42a5_bada_8126ed1944c3.slice/crio-dd6ecbf0825aefbc4b0fcc70386a85180ba331203f0e229d888867b6fa658c0e WatchSource:0}: Error finding container dd6ecbf0825aefbc4b0fcc70386a85180ba331203f0e229d888867b6fa658c0e: Status 404 returned error can't find the container with id dd6ecbf0825aefbc4b0fcc70386a85180ba331203f0e229d888867b6fa658c0e Jul 21 04:09:18.188615 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:18.188309 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6254d1d7_216d_4417_9fcb_6f71dc38129b.slice/crio-48a6d1617d66f8fc9a9de7fd3bd3cf09409ee00b081484f6f75456ff5b5d1cdc WatchSource:0}: Error finding container 48a6d1617d66f8fc9a9de7fd3bd3cf09409ee00b081484f6f75456ff5b5d1cdc: Status 404 returned error can't find the container with id 48a6d1617d66f8fc9a9de7fd3bd3cf09409ee00b081484f6f75456ff5b5d1cdc Jul 21 04:09:18.189698 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:18.189262 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab7d0e5_db4f_4988_80a9_4edbd26c174c.slice/crio-fcb6e8c20ea1863dc90c8715daca57782a8982135a914bbb25cdca6f057a8aeb WatchSource:0}: Error finding container fcb6e8c20ea1863dc90c8715daca57782a8982135a914bbb25cdca6f057a8aeb: Status 404 returned error can't find the container with id fcb6e8c20ea1863dc90c8715daca57782a8982135a914bbb25cdca6f057a8aeb Jul 21 04:09:18.193892 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:18.193463 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406658cc_5989_4a5e_8ef8_a27780de41a5.slice/crio-bc1979518c5d422b15b017e8f14862f94a53752a85c887e511d2dd0197de99af WatchSource:0}: Error finding container bc1979518c5d422b15b017e8f14862f94a53752a85c887e511d2dd0197de99af: Status 404 returned error can't find the container with id bc1979518c5d422b15b017e8f14862f94a53752a85c887e511d2dd0197de99af Jul 21 04:09:18.194415 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:18.194295 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5da763_6747_4693_af09_9370b50f6e56.slice/crio-1ab6bab4dd3a3ce6a2fd0b31f8a9f0b30641adbb644c234b97109b715cb778d6 WatchSource:0}: Error finding container 1ab6bab4dd3a3ce6a2fd0b31f8a9f0b30641adbb644c234b97109b715cb778d6: Status 404 returned error can't find the container with id 1ab6bab4dd3a3ce6a2fd0b31f8a9f0b30641adbb644c234b97109b715cb778d6 Jul 21 04:09:18.195666 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:18.195643 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda894722a_d3ee_4318_9673_6a1962e067e5.slice/crio-84c1352a2c5df79ae27b5914a5d95a72340b6afbfb2234a9af7444654d75c0f0 WatchSource:0}: Error finding container 84c1352a2c5df79ae27b5914a5d95a72340b6afbfb2234a9af7444654d75c0f0: Status 404 returned error can't find the container with id 84c1352a2c5df79ae27b5914a5d95a72340b6afbfb2234a9af7444654d75c0f0 Jul 21 04:09:18.264439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.264262 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:18.264586 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:18.264556 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:18.268710 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.268682 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" event={"ID":"3e5da763-6747-4693-af09-9370b50f6e56","Type":"ContainerStarted","Data":"1ab6bab4dd3a3ce6a2fd0b31f8a9f0b30641adbb644c234b97109b715cb778d6"} Jul 21 04:09:18.270256 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.270236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" event={"ID":"a894722a-d3ee-4318-9673-6a1962e067e5","Type":"ContainerStarted","Data":"84c1352a2c5df79ae27b5914a5d95a72340b6afbfb2234a9af7444654d75c0f0"} Jul 21 04:09:18.271438 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.271414 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" event={"ID":"241293ba-5312-43da-b1d6-7ba42c71cfbd","Type":"ContainerStarted","Data":"45557716ce669ff9768e607da8e307e477329bdc8c91d95cb3a568b567ac44e1"} Jul 21 04:09:18.273810 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.273787 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" event={"ID":"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c","Type":"ContainerStarted","Data":"24b40d3e105221b79f0b3e8884e7a0e6681863bc093d5acc10110ffafd00578b"} Jul 21 04:09:18.277518 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.277490 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-897jc" event={"ID":"af409d50-2522-42a5-bada-8126ed1944c3","Type":"ContainerStarted","Data":"dd6ecbf0825aefbc4b0fcc70386a85180ba331203f0e229d888867b6fa658c0e"} Jul 21 04:09:18.279304 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.279283 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mndqf" event={"ID":"3f65e21d-704f-40f0-b476-f76d71d79fff","Type":"ContainerStarted","Data":"4b724bc3d7e4e80748a3899ca1e5f75a18dcce99e086e4c2b99cb8cbc0da9b5e"} Jul 21 04:09:18.285940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.285887 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bkscj" event={"ID":"406658cc-5989-4a5e-8ef8-a27780de41a5","Type":"ContainerStarted","Data":"bc1979518c5d422b15b017e8f14862f94a53752a85c887e511d2dd0197de99af"} Jul 21 04:09:18.290709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.290687 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-snqhr" event={"ID":"5ab7d0e5-db4f-4988-80a9-4edbd26c174c","Type":"ContainerStarted","Data":"fcb6e8c20ea1863dc90c8715daca57782a8982135a914bbb25cdca6f057a8aeb"} Jul 21 04:09:18.293162 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.293144 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j676g" event={"ID":"6254d1d7-216d-4417-9fcb-6f71dc38129b","Type":"ContainerStarted","Data":"48a6d1617d66f8fc9a9de7fd3bd3cf09409ee00b081484f6f75456ff5b5d1cdc"} Jul 21 04:09:18.613155 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.613072 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:18.613155 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.613118 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:18.613318 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:18.613217 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:18.613318 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:18.613232 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:18.613318 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:18.613274 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs podName:fabd6623-e601-4828-8b83-673db7105de6 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:20.613257293 +0000 UTC m=+6.012590049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs") pod "network-metrics-daemon-8w86n" (UID: "fabd6623-e601-4828-8b83-673db7105de6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:18.613318 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:18.613288 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret podName:c21ab5db-d116-4b5f-8f61-df4fe3a0ef89 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:20.6132823 +0000 UTC m=+6.012615036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret") pod "global-pull-secret-syncer-wn4dg" (UID: "c21ab5db-d116-4b5f-8f61-df4fe3a0ef89") : object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:18.714103 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:18.714063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:18.714266 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:18.714227 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 04:09:18.714266 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:18.714253 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 04:09:18.714266 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:18.714266 2567 projected.go:196] Error preparing data for projected volume kube-api-access-mvwfz for pod openshift-network-diagnostics/network-check-target-ft8hx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:18.714432 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:18.714331 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz podName:3fff3e92-f836-43fd-8c86-f79e0712cdd1 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:20.714312777 +0000 UTC m=+6.113645518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvwfz" (UniqueName: "kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz") pod "network-check-target-ft8hx" (UID: "3fff3e92-f836-43fd-8c86-f79e0712cdd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:19.032118 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:19.032037 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 04:04:17 +0000 UTC" deadline="2028-04-16 20:14:31.249425194 +0000 UTC" Jul 21 04:09:19.032118 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:19.032106 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15256h5m12.217323863s" Jul 21 04:09:19.270769 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:19.270110 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:19.270769 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:19.270227 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:19.270769 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:19.270637 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:19.270769 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:19.270729 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:19.321578 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:19.321495 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" event={"ID":"3e07ae959d76e4e217922a691b6e844d","Type":"ContainerStarted","Data":"f96c823c10ba398796f0b0143cf28eced63e47fce5b7f699ec69be834176bb10"} Jul 21 04:09:19.348793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:19.348688 2567 generic.go:356] "Generic (PLEG): container finished" podID="f1941da6dbd8a4efb4a368091e4943a9" containerID="febb74adcf5e6a40155386cae3ad57182ba93613dc929f153d1ef23e508f5b28" exitCode=0 Jul 21 04:09:19.348793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:19.348740 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" event={"ID":"f1941da6dbd8a4efb4a368091e4943a9","Type":"ContainerDied","Data":"febb74adcf5e6a40155386cae3ad57182ba93613dc929f153d1ef23e508f5b28"} Jul 21 04:09:19.365921 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:19.365866 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-226.ec2.internal" podStartSLOduration=3.365849197 podStartE2EDuration="3.365849197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:18.298322098 +0000 UTC m=+3.697654866" watchObservedRunningTime="2026-07-21 04:09:19.365849197 +0000 UTC m=+4.765181956" Jul 21 04:09:20.260238 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:20.260210 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:20.260765 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:20.260367 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:20.377366 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:20.377108 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" event={"ID":"f1941da6dbd8a4efb4a368091e4943a9","Type":"ContainerStarted","Data":"ea8c9c7ac00c7438ed9e2432c93e40e5900be3bda41fc4671c2d2a3ff9eafd8f"} Jul 21 04:09:20.632966 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:20.631966 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:20.632966 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:20.632038 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:20.632966 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:20.632234 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:20.632966 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:20.632369 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs podName:fabd6623-e601-4828-8b83-673db7105de6 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:24.632288777 +0000 UTC m=+10.031621526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs") pod "network-metrics-daemon-8w86n" (UID: "fabd6623-e601-4828-8b83-673db7105de6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:20.632966 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:20.632783 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:20.632966 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:20.632877 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret podName:c21ab5db-d116-4b5f-8f61-df4fe3a0ef89 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:24.632861156 +0000 UTC m=+10.032193893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret") pod "global-pull-secret-syncer-wn4dg" (UID: "c21ab5db-d116-4b5f-8f61-df4fe3a0ef89") : object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:20.733307 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:20.733239 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:20.733582 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:20.733548 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 04:09:20.733582 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:20.733579 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 04:09:20.733582 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:20.733593 2567 projected.go:196] Error preparing data for projected volume kube-api-access-mvwfz for pod openshift-network-diagnostics/network-check-target-ft8hx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:20.733836 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:20.733686 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz podName:3fff3e92-f836-43fd-8c86-f79e0712cdd1 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:24.733664303 +0000 UTC m=+10.132997054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvwfz" (UniqueName: "kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz") pod "network-check-target-ft8hx" (UID: "3fff3e92-f836-43fd-8c86-f79e0712cdd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:21.264962 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:21.264899 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:21.265425 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:21.264911 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:21.265425 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:21.265044 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:21.265425 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:21.265168 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:22.261151 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:22.260775 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:22.261151 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:22.260914 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:23.260412 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:23.260383 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:23.260888 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:23.260427 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:23.260888 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:23.260514 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:23.260983 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:23.260913 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:24.259990 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:24.259956 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:24.260173 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:24.260112 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:24.665969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:24.665580 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:24.665969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:24.665672 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:24.665969 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:24.665797 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:24.665969 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:24.665833 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:24.665969 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:24.665862 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret podName:c21ab5db-d116-4b5f-8f61-df4fe3a0ef89 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:32.665844618 +0000 UTC m=+18.065177361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret") pod "global-pull-secret-syncer-wn4dg" (UID: "c21ab5db-d116-4b5f-8f61-df4fe3a0ef89") : object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:24.665969 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:24.665900 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs podName:fabd6623-e601-4828-8b83-673db7105de6 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:32.665882465 +0000 UTC m=+18.065215216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs") pod "network-metrics-daemon-8w86n" (UID: "fabd6623-e601-4828-8b83-673db7105de6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:24.766411 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:24.766369 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:24.766590 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:24.766556 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 04:09:24.766590 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:24.766584 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 04:09:24.766699 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:24.766597 2567 projected.go:196] Error preparing data for projected volume kube-api-access-mvwfz for pod openshift-network-diagnostics/network-check-target-ft8hx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:24.766699 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:24.766670 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz podName:3fff3e92-f836-43fd-8c86-f79e0712cdd1 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:32.766647961 +0000 UTC m=+18.165980698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvwfz" (UniqueName: "kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz") pod "network-check-target-ft8hx" (UID: "3fff3e92-f836-43fd-8c86-f79e0712cdd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:25.262297 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:25.261685 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:25.262297 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:25.261803 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:25.262297 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:25.262160 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:25.262297 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:25.262254 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:26.260277 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:26.260232 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:26.260791 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:26.260401 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:27.260601 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:27.260564 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:27.261041 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:27.260685 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:27.261041 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:27.260759 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:27.261041 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:27.260878 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:28.259936 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:28.259901 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:28.260119 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:28.260034 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:29.260615 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:29.260576 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:29.261058 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:29.260723 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:29.261058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:29.260779 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:29.261058 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:29.260895 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:30.260251 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:30.260214 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:30.260418 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:30.260382 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:31.260579 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:31.260532 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:31.261038 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:31.260532 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:31.261038 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:31.260677 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:31.261038 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:31.260785 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:32.260903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:32.260863 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:32.261400 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:32.261000 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:32.729655 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:32.729585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:32.729655 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:32.729639 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:32.729818 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:32.729717 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:32.729818 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:32.729725 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:32.729818 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:32.729767 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret podName:c21ab5db-d116-4b5f-8f61-df4fe3a0ef89 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:48.729753872 +0000 UTC m=+34.129086609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret") pod "global-pull-secret-syncer-wn4dg" (UID: "c21ab5db-d116-4b5f-8f61-df4fe3a0ef89") : object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:32.729818 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:32.729781 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs podName:fabd6623-e601-4828-8b83-673db7105de6 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:48.729774311 +0000 UTC m=+34.129107047 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs") pod "network-metrics-daemon-8w86n" (UID: "fabd6623-e601-4828-8b83-673db7105de6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:32.830804 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:32.830774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:32.830939 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:32.830914 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 04:09:32.830939 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:32.830932 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 04:09:32.831031 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:32.830945 2567 projected.go:196] Error preparing data for projected volume kube-api-access-mvwfz for pod openshift-network-diagnostics/network-check-target-ft8hx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:32.831031 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:32.831004 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz podName:3fff3e92-f836-43fd-8c86-f79e0712cdd1 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:48.830987318 +0000 UTC m=+34.230320056 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvwfz" (UniqueName: "kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz") pod "network-check-target-ft8hx" (UID: "3fff3e92-f836-43fd-8c86-f79e0712cdd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:33.260427 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:33.260392 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:33.260427 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:33.260420 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:33.260675 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:33.260518 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:33.260675 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:33.260647 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:34.260271 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:34.260236 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:34.260725 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:34.260371 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:35.261134 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:35.261101 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:35.261645 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:35.261183 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:35.261645 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:35.261216 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:35.261645 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:35.261263 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:36.259987 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:36.259950 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:36.260172 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:36.260094 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:37.260714 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.260687 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:37.261142 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:37.260815 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:37.261209 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.261178 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:37.261288 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:37.261263 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:37.414509 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.413759 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" event={"ID":"3e5da763-6747-4693-af09-9370b50f6e56","Type":"ContainerStarted","Data":"f2ede8b94308e2af7fb6f9eb7e2ac4cd647928c7c2cf58731567d5becaad7d28"} Jul 21 04:09:37.416786 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.415618 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" event={"ID":"a894722a-d3ee-4318-9673-6a1962e067e5","Type":"ContainerStarted","Data":"bfbfe0e7572eb9a905e9d5bd9ad83c2ca68c00225e027a6c1da9cf95af4d7858"} Jul 21 04:09:37.418874 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.418738 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" event={"ID":"241293ba-5312-43da-b1d6-7ba42c71cfbd","Type":"ContainerStarted","Data":"5335c6c963d1c9e6e87416cb9bfa92deda65ed3c3eb9237a76565a0e454e51b6"} Jul 21 04:09:37.418874 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.418817 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" event={"ID":"241293ba-5312-43da-b1d6-7ba42c71cfbd","Type":"ContainerStarted","Data":"55999d5586233026a0c86d314e3b8ebf4dedeea9c5706cfe9b42042aa0c6ba7d"} Jul 21 04:09:37.420435 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.420405 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" event={"ID":"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c","Type":"ContainerStarted","Data":"e7d2e1e5793b437ef5ef29d770b14212cb21ca24b22f6bbb4bb8ba60c39035f8"} Jul 21 04:09:37.421911 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.421869 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-897jc" event={"ID":"af409d50-2522-42a5-bada-8126ed1944c3","Type":"ContainerStarted","Data":"88b5b6bbd717500e056276af07c70cb275b44f7fa98f6513e30cbde2c3469503"} Jul 21 04:09:37.423510 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.423484 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bkscj" event={"ID":"406658cc-5989-4a5e-8ef8-a27780de41a5","Type":"ContainerStarted","Data":"edc8243562f596a989e55fbb9feeff10403bcdb4356d11cb7e1625336e345620"} Jul 21 04:09:37.424692 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.424668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-snqhr" event={"ID":"5ab7d0e5-db4f-4988-80a9-4edbd26c174c","Type":"ContainerStarted","Data":"d7da5f7797b3c7cc289a822e11b401c1ba5c3d4ca71c1bc29652d993fca2121d"} Jul 21 04:09:37.426229 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.426205 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j676g" event={"ID":"6254d1d7-216d-4417-9fcb-6f71dc38129b","Type":"ContainerStarted","Data":"e069c6e7b813977797afc0f62c6b9da5108158989d650d083ad671e46a86dc97"} Jul 21 04:09:37.438313 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.438270 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-226.ec2.internal" podStartSLOduration=21.43825743 podStartE2EDuration="21.43825743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:20.393924857 +0000 UTC m=+5.793257616" watchObservedRunningTime="2026-07-21 04:09:37.43825743 +0000 UTC m=+22.837590187" Jul 21 04:09:37.470994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.470942 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bkscj" podStartSLOduration=3.519884394 podStartE2EDuration="22.470921353s" totalImagesPullingTime="18.951036959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:37.470528122 +0000 UTC m=+22.869860881" watchObservedRunningTime="2026-07-21 04:09:37.470921353 +0000 UTC m=+22.870254113" Jul 21 04:09:37.471079 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.471056 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-897jc" podStartSLOduration=3.4610786940000002 podStartE2EDuration="22.471045277s" totalImagesPullingTime="19.009966583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:37.455237147 +0000 UTC m=+22.854569905" watchObservedRunningTime="2026-07-21 04:09:37.471045277 +0000 UTC m=+22.870378066" Jul 21 04:09:37.502240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.502201 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j676g" podStartSLOduration=3.546428518 podStartE2EDuration="22.502186527s" totalImagesPullingTime="18.955758009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:37.485515383 +0000 UTC m=+22.884848141" watchObservedRunningTime="2026-07-21 04:09:37.502186527 +0000 UTC m=+22.901519286" Jul 21 04:09:37.502533 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.502512 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2jgjj" podStartSLOduration=3.555079433 podStartE2EDuration="22.502505979s" totalImagesPullingTime="18.947426546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:37.502454493 +0000 UTC m=+22.901787251" watchObservedRunningTime="2026-07-21 04:09:37.502505979 +0000 UTC m=+22.901838774" Jul 21 04:09:37.517360 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:37.517309 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-snqhr" podStartSLOduration=3.560579433 podStartE2EDuration="22.517296267s" totalImagesPullingTime="18.956716834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:37.51693293 +0000 UTC m=+22.916265688" watchObservedRunningTime="2026-07-21 04:09:37.517296267 +0000 UTC m=+22.916629024" Jul 21 04:09:38.260761 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:38.260720 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:38.261675 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:38.260867 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:38.429540 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:38.429510 2567 generic.go:356] "Generic (PLEG): container finished" podID="3e5da763-6747-4693-af09-9370b50f6e56" containerID="f2ede8b94308e2af7fb6f9eb7e2ac4cd647928c7c2cf58731567d5becaad7d28" exitCode=0 Jul 21 04:09:38.429753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:38.429585 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" event={"ID":"3e5da763-6747-4693-af09-9370b50f6e56","Type":"ContainerDied","Data":"f2ede8b94308e2af7fb6f9eb7e2ac4cd647928c7c2cf58731567d5becaad7d28"} Jul 21 04:09:38.432922 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:38.432893 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" event={"ID":"241293ba-5312-43da-b1d6-7ba42c71cfbd","Type":"ContainerStarted","Data":"153e4409aa26ad1f9d732db50c0fce143d5bb9ab9b89ae9566cb31be2537f5ab"} Jul 21 04:09:38.433019 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:38.432959 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" event={"ID":"241293ba-5312-43da-b1d6-7ba42c71cfbd","Type":"ContainerStarted","Data":"496fb3b2bcd9ddb9343f9d62dcc816f79a75db4caf037c95493ed69204b361ce"} Jul 21 04:09:38.433019 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:38.432990 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" event={"ID":"241293ba-5312-43da-b1d6-7ba42c71cfbd","Type":"ContainerStarted","Data":"791c6553d24a794f9b2ef1a1becddd4862b348875182849f7d7ca5e7e884818b"} Jul 21 04:09:38.433019 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:38.433016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" event={"ID":"241293ba-5312-43da-b1d6-7ba42c71cfbd","Type":"ContainerStarted","Data":"5a37180a07179696520dc9f219d1a47505d1726322f46b20a5d956573522e32d"} Jul 21 04:09:38.731813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:38.731792 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 04:09:39.199099 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:39.198945 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T04:09:38.731806974Z","UUID":"4c793220-0dc3-41ee-91a5-2d07e3f1cf6c","Handler":null,"Name":"","Endpoint":""} Jul 21 04:09:39.200880 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:39.200851 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 04:09:39.201024 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:39.200903 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 04:09:39.260934 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:39.260897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:39.260934 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:39.260921 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:39.261425 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:39.261021 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:39.261425 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:39.261128 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:39.437121 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:39.437083 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" event={"ID":"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c","Type":"ContainerStarted","Data":"fe84fa1b2db432915fbe29a3287fc3d5ca18d900a0e8e8a24d562c358637970a"} Jul 21 04:09:39.438818 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:39.438789 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mndqf" event={"ID":"3f65e21d-704f-40f0-b476-f76d71d79fff","Type":"ContainerStarted","Data":"dc935525f78e0b32a6c54af4b46010f5b6111aa48f3f16f1ee41223b394b4bec"} Jul 21 04:09:39.454173 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:39.453866 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mndqf" podStartSLOduration=5.464617671 podStartE2EDuration="24.453851481s" totalImagesPullingTime="18.98923381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:39.45312554 +0000 UTC m=+24.852458299" watchObservedRunningTime="2026-07-21 04:09:39.453851481 +0000 UTC m=+24.853184239" Jul 21 04:09:40.260855 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:40.260665 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:40.261000 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:40.260952 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:40.443212 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:40.443174 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" event={"ID":"3d359fcb-2a3e-4859-a07c-1a8ceb9d904c","Type":"ContainerStarted","Data":"70d3729078592eb51b27c59cd25e4fd67e225bdf8683404a7d7f9dc3a1241c1d"} Jul 21 04:09:40.446935 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:40.446905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" event={"ID":"241293ba-5312-43da-b1d6-7ba42c71cfbd","Type":"ContainerStarted","Data":"4f3793845f8b70db4bb8d0a24571078c94b539a66adb4522b705ac263e97494e"} Jul 21 04:09:41.260350 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:41.260304 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:41.260616 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:41.260454 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:41.260616 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:41.260544 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:41.260723 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:41.260659 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:42.023779 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:42.023741 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:42.024422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:42.024402 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:42.040189 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:42.040139 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qw7df" podStartSLOduration=5.325754392 podStartE2EDuration="27.040120327s" totalImagesPullingTime="21.714365935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:40.460636163 +0000 UTC m=+25.859968921" watchObservedRunningTime="2026-07-21 04:09:42.040120327 +0000 UTC m=+27.439453086" Jul 21 04:09:42.260073 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:42.260036 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:42.260239 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:42.260177 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:42.452270 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:42.452201 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:42.452763 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:42.452742 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j676g" Jul 21 04:09:43.260815 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:43.260640 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:43.261357 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:43.260694 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:43.261357 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:43.260886 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:43.261357 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:43.260971 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:43.455795 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:43.455760 2567 generic.go:356] "Generic (PLEG): container finished" podID="3e5da763-6747-4693-af09-9370b50f6e56" containerID="866b6047774d0eed5f3e7a63d3b48eb70efb2e4e8594d3366b6b74c067f844a8" exitCode=0 Jul 21 04:09:43.455970 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:43.455805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" event={"ID":"3e5da763-6747-4693-af09-9370b50f6e56","Type":"ContainerDied","Data":"866b6047774d0eed5f3e7a63d3b48eb70efb2e4e8594d3366b6b74c067f844a8"} Jul 21 04:09:43.459614 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:43.459586 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" event={"ID":"241293ba-5312-43da-b1d6-7ba42c71cfbd","Type":"ContainerStarted","Data":"df31b6872494d1b924f1f8dc1f1b134020529e9857de9b228a24b195a6614f6f"} Jul 21 04:09:43.501592 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:43.501549 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" podStartSLOduration=9.447619549 podStartE2EDuration="28.501532953s" totalImagesPullingTime="19.053913404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:43.500331251 +0000 UTC m=+28.899664009" watchObservedRunningTime="2026-07-21 04:09:43.501532953 +0000 UTC m=+28.900865713" Jul 21 04:09:44.260717 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.260686 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:44.260929 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:44.260798 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:44.463399 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.463373 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:44.463682 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.463456 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:44.463682 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.463509 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:44.480100 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.480074 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:44.480889 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.480866 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:09:44.912208 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.912124 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ft8hx"] Jul 21 04:09:44.912365 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.912258 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:44.912365 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:44.912332 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:44.917011 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.916982 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wn4dg"] Jul 21 04:09:44.917153 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.917096 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:44.917219 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:44.917182 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:44.917636 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.917613 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8w86n"] Jul 21 04:09:44.917752 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:44.917739 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:44.917846 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:44.917827 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:45.466779 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:45.466601 2567 generic.go:356] "Generic (PLEG): container finished" podID="3e5da763-6747-4693-af09-9370b50f6e56" containerID="33466a53a4e7734a7f7afa12754242a2c78928436e9c37259ceb1432df0e4e85" exitCode=0 Jul 21 04:09:45.466779 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:45.466681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" event={"ID":"3e5da763-6747-4693-af09-9370b50f6e56","Type":"ContainerDied","Data":"33466a53a4e7734a7f7afa12754242a2c78928436e9c37259ceb1432df0e4e85"} Jul 21 04:09:46.260518 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:46.260483 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:46.260664 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:46.260483 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:46.260664 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:46.260615 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:46.260664 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:46.260487 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:46.260782 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:46.260666 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:46.260782 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:46.260735 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:47.473880 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:47.473843 2567 generic.go:356] "Generic (PLEG): container finished" podID="3e5da763-6747-4693-af09-9370b50f6e56" containerID="11e601c60a596ce5477cc756546591ac96484e1b5f0b09fc78148034fc890a71" exitCode=0 Jul 21 04:09:47.474565 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:47.473901 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" event={"ID":"3e5da763-6747-4693-af09-9370b50f6e56","Type":"ContainerDied","Data":"11e601c60a596ce5477cc756546591ac96484e1b5f0b09fc78148034fc890a71"} Jul 21 04:09:48.260790 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:48.260756 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:48.260978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:48.260870 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:48.260978 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.260876 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ft8hx" podUID="3fff3e92-f836-43fd-8c86-f79e0712cdd1" Jul 21 04:09:48.260978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:48.260880 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:48.261127 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.260976 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn4dg" podUID="c21ab5db-d116-4b5f-8f61-df4fe3a0ef89" Jul 21 04:09:48.261127 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.261060 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w86n" podUID="fabd6623-e601-4828-8b83-673db7105de6" Jul 21 04:09:48.747235 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:48.747149 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:48.747235 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:48.747213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:48.747734 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.747300 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:48.747734 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.747328 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:48.747734 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.747405 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret podName:c21ab5db-d116-4b5f-8f61-df4fe3a0ef89 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:20.74738041 +0000 UTC m=+66.146713147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret") pod "global-pull-secret-syncer-wn4dg" (UID: "c21ab5db-d116-4b5f-8f61-df4fe3a0ef89") : object "kube-system"/"original-pull-secret" not registered Jul 21 04:09:48.747734 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.747426 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs podName:fabd6623-e601-4828-8b83-673db7105de6 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:20.747417365 +0000 UTC m=+66.146750103 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs") pod "network-metrics-daemon-8w86n" (UID: "fabd6623-e601-4828-8b83-673db7105de6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 04:09:48.848166 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:48.848134 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:48.848353 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.848285 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 04:09:48.848353 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.848304 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 04:09:48.848353 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.848313 2567 projected.go:196] Error preparing data for projected volume kube-api-access-mvwfz for pod openshift-network-diagnostics/network-check-target-ft8hx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:48.848514 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:48.848389 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz podName:3fff3e92-f836-43fd-8c86-f79e0712cdd1 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:20.848371709 +0000 UTC m=+66.247704465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mvwfz" (UniqueName: "kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz") pod "network-check-target-ft8hx" (UID: "3fff3e92-f836-43fd-8c86-f79e0712cdd1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 04:09:49.877649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.877619 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-226.ec2.internal" event="NodeReady" Jul 21 04:09:49.878186 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.877759 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 04:09:49.917278 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.916002 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fbfd644f8-thjd2"] Jul 21 04:09:49.918952 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.918902 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-7r77f"] Jul 21 04:09:49.919084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.919052 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:49.921639 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.921614 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c"] Jul 21 04:09:49.921757 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.921668 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 04:09:49.921757 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.921724 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 04:09:49.921757 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.921726 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jngx5\"" Jul 21 04:09:49.921917 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.921834 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 04:09:49.921975 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.921961 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:49.923562 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.923539 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k"] Jul 21 04:09:49.923680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.923635 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:49.925072 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.925052 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 04:09:49.925432 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.925327 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 04:09:49.925714 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.925668 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dppvf\"" Jul 21 04:09:49.925811 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.925785 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.926137 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.925963 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.926490 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.926470 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5"] Jul 21 04:09:49.926594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.926539 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:49.928305 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.928284 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4brc7\"" Jul 21 04:09:49.928524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.928492 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:49.929313 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.929291 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 21 04:09:49.929477 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.929355 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 21 04:09:49.929549 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.929388 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.929604 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.929401 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.930737 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.930662 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 21 04:09:49.931043 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.930662 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.931043 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.930973 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 04:09:49.931169 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.931150 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-69jxl\"" Jul 21 04:09:49.931865 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.931845 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.932267 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.932137 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.932267 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.932173 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.932267 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.932250 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 04:09:49.932630 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.932542 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 04:09:49.932630 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.932567 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m899n\"" Jul 21 04:09:49.933521 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.933503 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 04:09:49.936617 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.936601 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fbfd644f8-thjd2"] Jul 21 04:09:49.936786 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.936772 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-7r77f"] Jul 21 04:09:49.936891 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.936880 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27"] Jul 21 04:09:49.939178 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.939157 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k"] Jul 21 04:09:49.939280 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.939210 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6"] Jul 21 04:09:49.940275 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.940248 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27" Jul 21 04:09:49.941844 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.941825 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-g9hm8"] Jul 21 04:09:49.941963 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.941943 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:49.943555 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.943098 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.943555 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.943278 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.943555 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.943383 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-h52ns\"" Jul 21 04:09:49.945383 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.945366 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp"] Jul 21 04:09:49.946515 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.946494 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 04:09:49.946611 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.946568 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7d4m6\"" Jul 21 04:09:49.946669 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.946607 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.946788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.946767 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 04:09:49.946935 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.946915 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.946997 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.946947 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:49.948150 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.948134 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c6fb748b6-ncql2"] Jul 21 04:09:49.949791 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.949775 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:49.949925 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.949904 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 21 04:09:49.950071 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.950042 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 21 04:09:49.950247 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.949992 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.950402 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.949875 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.950768 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.950747 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-t8jlf\"" Jul 21 04:09:49.950897 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.950833 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5"] Jul 21 04:09:49.950897 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.950865 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp"] Jul 21 04:09:49.951266 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.951231 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:49.953900 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.953878 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 21 04:09:49.954881 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.954847 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c"] Jul 21 04:09:49.955075 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.955059 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hwcbj"] Jul 21 04:09:49.956218 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.956197 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp" Jul 21 04:09:49.959292 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.958859 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 21 04:09:49.959400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.959361 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dw6n5\"" Jul 21 04:09:49.959602 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.959566 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8q48j\"" Jul 21 04:09:49.959707 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.959692 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 21 04:09:49.960129 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.960106 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.960359 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.960319 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.960625 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.960608 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 21 04:09:49.960728 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.960692 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 21 04:09:49.960793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.960768 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 21 04:09:49.960847 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.960639 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.960847 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.960616 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 21 04:09:49.960954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.960667 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.961797 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.961781 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wx6xf"] Jul 21 04:09:49.962089 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.962063 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwcbj" Jul 21 04:09:49.964606 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.964574 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 04:09:49.964704 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.964681 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2pdlm\"" Jul 21 04:09:49.965165 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965055 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 04:09:49.965165 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965090 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 04:09:49.965318 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965231 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zksv2\"" Jul 21 04:09:49.965416 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965392 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27"] Jul 21 04:09:49.965507 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965427 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwcbj"] Jul 21 04:09:49.965507 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965454 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp"] Jul 21 04:09:49.965507 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965477 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-g9hm8"] Jul 21 04:09:49.965507 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965495 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wx6xf"] Jul 21 04:09:49.965507 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965503 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:49.965746 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.965515 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c6fb748b6-ncql2"] Jul 21 04:09:49.967841 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.967821 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b7grb\"" Jul 21 04:09:49.967920 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.967811 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 04:09:49.967997 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.967926 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 04:09:49.973177 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.972978 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6"] Jul 21 04:09:49.974838 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:49.974816 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp"] Jul 21 04:09:50.058318 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058282 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d55741-8411-4b6b-91fd-599aeb651f83-tmp\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.058462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97374277-5b6f-4196-b591-b152148b4b33-trusted-ca\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.058462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058385 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70bac8e0-64e1-437e-878e-d67cf04577ce-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:50.058462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlldb\" (UniqueName: \"kubernetes.io/projected/e37a0b54-1b3b-4899-b7cc-bb6295abd52c-kube-api-access-mlldb\") pod \"network-check-source-7bc5b6bf85-ln2tp\" (UID: \"e37a0b54-1b3b-4899-b7cc-bb6295abd52c\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp" Jul 21 04:09:50.058601 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058472 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klh2h\" (UniqueName: \"kubernetes.io/projected/8f8543d9-2530-4536-b660-85e318bf3d60-kube-api-access-klh2h\") pod \"ingress-canary-hwcbj\" (UID: \"8f8543d9-2530-4536-b660-85e318bf3d60\") " pod="openshift-ingress-canary/ingress-canary-hwcbj" Jul 21 04:09:50.058601 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058499 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97374277-5b6f-4196-b591-b152148b4b33-serving-cert\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.058601 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058529 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4297ba8e-5cf6-40a5-90ba-2d1215b9237f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-zntm6\" (UID: \"4297ba8e-5cf6-40a5-90ba-2d1215b9237f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.058601 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058568 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-trusted-ca\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.058752 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058602 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d55741-8411-4b6b-91fd-599aeb651f83-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.058752 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058632 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.058752 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058659 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-installation-pull-secrets\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.058752 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058684 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97374277-5b6f-4196-b591-b152148b4b33-config\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.058752 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058711 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-default-certificate\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.058752 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058737 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9lz8\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-kube-api-access-s9lz8\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.058954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058761 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:50.058954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058793 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf7q7\" (UniqueName: \"kubernetes.io/projected/19d55741-8411-4b6b-91fd-599aeb651f83-kube-api-access-hf7q7\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.058954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058816 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/564513d6-3759-4aa0-9318-bc34a30a6bbb-config-volume\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.058954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058837 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/19d55741-8411-4b6b-91fd-599aeb651f83-snapshots\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.058954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058859 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564513d6-3759-4aa0-9318-bc34a30a6bbb-tmp-dir\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.058954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058918 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-image-registry-private-configuration\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.059192 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058955 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6317ead8-1226-4ac5-ac8a-89e094b02e72-serving-cert\") pod \"service-ca-operator-5bcbc94c78-ccdv5\" (UID: \"6317ead8-1226-4ac5-ac8a-89e094b02e72\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.059192 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058978 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.059192 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.058994 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:50.059192 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059011 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdff\" (UniqueName: \"kubernetes.io/projected/4297ba8e-5cf6-40a5-90ba-2d1215b9237f-kube-api-access-jzdff\") pod \"kube-storage-version-migrator-operator-7658d9b749-zntm6\" (UID: \"4297ba8e-5cf6-40a5-90ba-2d1215b9237f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.059192 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059038 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t76tf\" (UniqueName: \"kubernetes.io/projected/564513d6-3759-4aa0-9318-bc34a30a6bbb-kube-api-access-t76tf\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.059192 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059089 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f8543d9-2530-4536-b660-85e318bf3d60-cert\") pod \"ingress-canary-hwcbj\" (UID: \"8f8543d9-2530-4536-b660-85e318bf3d60\") " pod="openshift-ingress-canary/ingress-canary-hwcbj" Jul 21 04:09:50.059192 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19d55741-8411-4b6b-91fd-599aeb651f83-serving-cert\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059194 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxxw5\" (UniqueName: \"kubernetes.io/projected/6889f625-36b3-431e-9b82-c71e379c1cee-kube-api-access-gxxw5\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059222 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d55741-8411-4b6b-91fd-599aeb651f83-service-ca-bundle\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059246 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059269 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fjv\" (UniqueName: \"kubernetes.io/projected/176aa68a-fa60-43e4-9f48-229ca8a92394-kube-api-access-58fjv\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059315 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-bound-sa-token\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059357 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059386 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkkw4\" (UniqueName: \"kubernetes.io/projected/d05da316-6261-4ce7-9c1e-53ffa7ff58bc-kube-api-access-xkkw4\") pod \"volume-data-source-validator-78cf4c6f6-4nc27\" (UID: \"d05da316-6261-4ce7-9c1e-53ffa7ff58bc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6889f625-36b3-431e-9b82-c71e379c1cee-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6317ead8-1226-4ac5-ac8a-89e094b02e72-config\") pod \"service-ca-operator-5bcbc94c78-ccdv5\" (UID: \"6317ead8-1226-4ac5-ac8a-89e094b02e72\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-stats-auth\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059515 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-certificates\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059533 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lsr8\" (UniqueName: \"kubernetes.io/projected/6317ead8-1226-4ac5-ac8a-89e094b02e72-kube-api-access-2lsr8\") pod \"service-ca-operator-5bcbc94c78-ccdv5\" (UID: \"6317ead8-1226-4ac5-ac8a-89e094b02e72\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.059550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059549 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7hb\" (UniqueName: \"kubernetes.io/projected/ded6b4a0-3715-4e78-8bce-7505e977b0aa-kube-api-access-rg7hb\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.060000 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059570 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7798p\" (UniqueName: \"kubernetes.io/projected/97374277-5b6f-4196-b591-b152148b4b33-kube-api-access-7798p\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.060000 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.060000 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059618 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-ca-trust-extracted\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.060000 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.059660 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4297ba8e-5cf6-40a5-90ba-2d1215b9237f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-zntm6\" (UID: \"4297ba8e-5cf6-40a5-90ba-2d1215b9237f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.108629 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.108595 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:50.160412 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7798p\" (UniqueName: \"kubernetes.io/projected/97374277-5b6f-4196-b591-b152148b4b33-kube-api-access-7798p\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.160412 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.160649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160442 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-ca-trust-extracted\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.160649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160467 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4297ba8e-5cf6-40a5-90ba-2d1215b9237f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-zntm6\" (UID: \"4297ba8e-5cf6-40a5-90ba-2d1215b9237f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.160649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160510 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d55741-8411-4b6b-91fd-599aeb651f83-tmp\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.160649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160542 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97374277-5b6f-4196-b591-b152148b4b33-trusted-ca\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.160649 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.160550 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 04:09:50.160649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70bac8e0-64e1-437e-878e-d67cf04577ce-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:50.160649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160592 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlldb\" (UniqueName: \"kubernetes.io/projected/e37a0b54-1b3b-4899-b7cc-bb6295abd52c-kube-api-access-mlldb\") pod \"network-check-source-7bc5b6bf85-ln2tp\" (UID: \"e37a0b54-1b3b-4899-b7cc-bb6295abd52c\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp" Jul 21 04:09:50.160649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160620 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klh2h\" (UniqueName: \"kubernetes.io/projected/8f8543d9-2530-4536-b660-85e318bf3d60-kube-api-access-klh2h\") pod \"ingress-canary-hwcbj\" (UID: \"8f8543d9-2530-4536-b660-85e318bf3d60\") " pod="openshift-ingress-canary/ingress-canary-hwcbj" Jul 21 04:09:50.160649 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.160635 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls podName:564513d6-3759-4aa0-9318-bc34a30a6bbb nodeName:}" failed. No retries permitted until 2026-07-21 04:09:50.660614933 +0000 UTC m=+36.059947689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls") pod "dns-default-wx6xf" (UID: "564513d6-3759-4aa0-9318-bc34a30a6bbb") : secret "dns-default-metrics-tls" not found Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160670 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97374277-5b6f-4196-b591-b152148b4b33-serving-cert\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160715 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4297ba8e-5cf6-40a5-90ba-2d1215b9237f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-zntm6\" (UID: \"4297ba8e-5cf6-40a5-90ba-2d1215b9237f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-trusted-ca\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160776 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d55741-8411-4b6b-91fd-599aeb651f83-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160800 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160834 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-installation-pull-secrets\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160860 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97374277-5b6f-4196-b591-b152148b4b33-config\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-default-certificate\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160900 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-ca-trust-extracted\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160910 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9lz8\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-kube-api-access-s9lz8\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.160990 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:50.161035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7q7\" (UniqueName: \"kubernetes.io/projected/19d55741-8411-4b6b-91fd-599aeb651f83-kube-api-access-hf7q7\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161048 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/564513d6-3759-4aa0-9318-bc34a30a6bbb-config-volume\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161153 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/19d55741-8411-4b6b-91fd-599aeb651f83-snapshots\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161165 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d55741-8411-4b6b-91fd-599aeb651f83-tmp\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564513d6-3759-4aa0-9318-bc34a30a6bbb-tmp-dir\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161245 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-image-registry-private-configuration\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161294 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6317ead8-1226-4ac5-ac8a-89e094b02e72-serving-cert\") pod \"service-ca-operator-5bcbc94c78-ccdv5\" (UID: \"6317ead8-1226-4ac5-ac8a-89e094b02e72\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161364 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161424 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdff\" (UniqueName: \"kubernetes.io/projected/4297ba8e-5cf6-40a5-90ba-2d1215b9237f-kube-api-access-jzdff\") pod \"kube-storage-version-migrator-operator-7658d9b749-zntm6\" (UID: \"4297ba8e-5cf6-40a5-90ba-2d1215b9237f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161466 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t76tf\" (UniqueName: \"kubernetes.io/projected/564513d6-3759-4aa0-9318-bc34a30a6bbb-kube-api-access-t76tf\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f8543d9-2530-4536-b660-85e318bf3d60-cert\") pod \"ingress-canary-hwcbj\" (UID: \"8f8543d9-2530-4536-b660-85e318bf3d60\") " pod="openshift-ingress-canary/ingress-canary-hwcbj" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19d55741-8411-4b6b-91fd-599aeb651f83-serving-cert\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161544 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/70bac8e0-64e1-437e-878e-d67cf04577ce-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:50.161603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161566 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxxw5\" (UniqueName: \"kubernetes.io/projected/6889f625-36b3-431e-9b82-c71e379c1cee-kube-api-access-gxxw5\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161614 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d55741-8411-4b6b-91fd-599aeb651f83-service-ca-bundle\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.161633 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161656 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97374277-5b6f-4196-b591-b152148b4b33-trusted-ca\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161689 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fjv\" (UniqueName: \"kubernetes.io/projected/176aa68a-fa60-43e4-9f48-229ca8a92394-kube-api-access-58fjv\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161701 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/564513d6-3759-4aa0-9318-bc34a30a6bbb-config-volume\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161727 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-bound-sa-token\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161768 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161768 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/19d55741-8411-4b6b-91fd-599aeb651f83-snapshots\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.161783 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:09:50.661766967 +0000 UTC m=+36.061099718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : configmap references non-existent config key: service-ca.crt Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161819 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkkw4\" (UniqueName: \"kubernetes.io/projected/d05da316-6261-4ce7-9c1e-53ffa7ff58bc-kube-api-access-xkkw4\") pod \"volume-data-source-validator-78cf4c6f6-4nc27\" (UID: \"d05da316-6261-4ce7-9c1e-53ffa7ff58bc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161848 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6889f625-36b3-431e-9b82-c71e379c1cee-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.161854 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161877 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6317ead8-1226-4ac5-ac8a-89e094b02e72-config\") pod \"service-ca-operator-5bcbc94c78-ccdv5\" (UID: \"6317ead8-1226-4ac5-ac8a-89e094b02e72\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.161907 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:09:50.661894736 +0000 UTC m=+36.061227479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : secret "router-metrics-certs-default" not found Jul 21 04:09:50.162205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161935 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-stats-auth\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.161978 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-certificates\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.161999 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.162004 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lsr8\" (UniqueName: \"kubernetes.io/projected/6317ead8-1226-4ac5-ac8a-89e094b02e72-kube-api-access-2lsr8\") pod \"service-ca-operator-5bcbc94c78-ccdv5\" (UID: \"6317ead8-1226-4ac5-ac8a-89e094b02e72\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.162031 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7hb\" (UniqueName: \"kubernetes.io/projected/ded6b4a0-3715-4e78-8bce-7505e977b0aa-kube-api-access-rg7hb\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.162090 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls podName:6889f625-36b3-431e-9b82-c71e379c1cee nodeName:}" failed. No retries permitted until 2026-07-21 04:09:50.662074833 +0000 UTC m=+36.061407589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zzs2c" (UID: "6889f625-36b3-431e-9b82-c71e379c1cee") : secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.162096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-trusted-ca\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.162187 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.162201 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbfd644f8-thjd2: secret "image-registry-tls" not found Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.162260 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls podName:bfda70bf-8536-4acd-a548-fa8edd8b5ba8 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:50.662246454 +0000 UTC m=+36.061579204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls") pod "image-registry-5fbfd644f8-thjd2" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8") : secret "image-registry-tls" not found Jul 21 04:09:50.162868 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.162434 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/564513d6-3759-4aa0-9318-bc34a30a6bbb-tmp-dir\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.163413 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.163014 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4297ba8e-5cf6-40a5-90ba-2d1215b9237f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-zntm6\" (UID: \"4297ba8e-5cf6-40a5-90ba-2d1215b9237f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.163413 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.163316 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6317ead8-1226-4ac5-ac8a-89e094b02e72-config\") pod \"service-ca-operator-5bcbc94c78-ccdv5\" (UID: \"6317ead8-1226-4ac5-ac8a-89e094b02e72\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.163413 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.163371 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/97374277-5b6f-4196-b591-b152148b4b33-config\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.164156 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.164132 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d55741-8411-4b6b-91fd-599aeb651f83-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.164510 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.164491 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 04:09:50.164596 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.164545 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls podName:176aa68a-fa60-43e4-9f48-229ca8a92394 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:50.664529162 +0000 UTC m=+36.063861914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jms6k" (UID: "176aa68a-fa60-43e4-9f48-229ca8a92394") : secret "samples-operator-tls" not found Jul 21 04:09:50.164596 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.164587 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-certificates\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.164718 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.164677 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert podName:70bac8e0-64e1-437e-878e-d67cf04577ce nodeName:}" failed. No retries permitted until 2026-07-21 04:09:50.664660235 +0000 UTC m=+36.063992978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-6b4vp" (UID: "70bac8e0-64e1-437e-878e-d67cf04577ce") : secret "networking-console-plugin-cert" not found Jul 21 04:09:50.165018 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.164994 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d55741-8411-4b6b-91fd-599aeb651f83-service-ca-bundle\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.165256 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.165231 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6889f625-36b3-431e-9b82-c71e379c1cee-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:50.166129 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.166087 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4297ba8e-5cf6-40a5-90ba-2d1215b9237f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-zntm6\" (UID: \"4297ba8e-5cf6-40a5-90ba-2d1215b9237f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.166946 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.166923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-installation-pull-secrets\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.167167 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.167137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6317ead8-1226-4ac5-ac8a-89e094b02e72-serving-cert\") pod \"service-ca-operator-5bcbc94c78-ccdv5\" (UID: \"6317ead8-1226-4ac5-ac8a-89e094b02e72\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.167630 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.167584 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f8543d9-2530-4536-b660-85e318bf3d60-cert\") pod \"ingress-canary-hwcbj\" (UID: \"8f8543d9-2530-4536-b660-85e318bf3d60\") " pod="openshift-ingress-canary/ingress-canary-hwcbj" Jul 21 04:09:50.167709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.167689 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-default-certificate\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.168024 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.167979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/97374277-5b6f-4196-b591-b152148b4b33-serving-cert\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.168111 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.168043 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-stats-auth\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.168160 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.168120 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-image-registry-private-configuration\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.168795 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.168773 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19d55741-8411-4b6b-91fd-599aeb651f83-serving-cert\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.177321 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.177276 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7798p\" (UniqueName: \"kubernetes.io/projected/97374277-5b6f-4196-b591-b152148b4b33-kube-api-access-7798p\") pod \"console-operator-565b7b569d-g9hm8\" (UID: \"97374277-5b6f-4196-b591-b152148b4b33\") " pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.178753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.178707 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klh2h\" (UniqueName: \"kubernetes.io/projected/8f8543d9-2530-4536-b660-85e318bf3d60-kube-api-access-klh2h\") pod \"ingress-canary-hwcbj\" (UID: \"8f8543d9-2530-4536-b660-85e318bf3d60\") " pod="openshift-ingress-canary/ingress-canary-hwcbj" Jul 21 04:09:50.179828 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.179803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9lz8\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-kube-api-access-s9lz8\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.181448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.180808 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lsr8\" (UniqueName: \"kubernetes.io/projected/6317ead8-1226-4ac5-ac8a-89e094b02e72-kube-api-access-2lsr8\") pod \"service-ca-operator-5bcbc94c78-ccdv5\" (UID: \"6317ead8-1226-4ac5-ac8a-89e094b02e72\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.181448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.181408 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxxw5\" (UniqueName: \"kubernetes.io/projected/6889f625-36b3-431e-9b82-c71e379c1cee-kube-api-access-gxxw5\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:50.182070 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.182029 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-bound-sa-token\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.182167 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.182144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7q7\" (UniqueName: \"kubernetes.io/projected/19d55741-8411-4b6b-91fd-599aeb651f83-kube-api-access-hf7q7\") pod \"insights-operator-555d9d7885-7r77f\" (UID: \"19d55741-8411-4b6b-91fd-599aeb651f83\") " pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.182386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.182323 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7hb\" (UniqueName: \"kubernetes.io/projected/ded6b4a0-3715-4e78-8bce-7505e977b0aa-kube-api-access-rg7hb\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.183234 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.182966 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdff\" (UniqueName: \"kubernetes.io/projected/4297ba8e-5cf6-40a5-90ba-2d1215b9237f-kube-api-access-jzdff\") pod \"kube-storage-version-migrator-operator-7658d9b749-zntm6\" (UID: \"4297ba8e-5cf6-40a5-90ba-2d1215b9237f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.183234 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.183166 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fjv\" (UniqueName: \"kubernetes.io/projected/176aa68a-fa60-43e4-9f48-229ca8a92394-kube-api-access-58fjv\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:50.183442 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.183419 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkkw4\" (UniqueName: \"kubernetes.io/projected/d05da316-6261-4ce7-9c1e-53ffa7ff58bc-kube-api-access-xkkw4\") pod \"volume-data-source-validator-78cf4c6f6-4nc27\" (UID: \"d05da316-6261-4ce7-9c1e-53ffa7ff58bc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27" Jul 21 04:09:50.183589 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.183561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlldb\" (UniqueName: \"kubernetes.io/projected/e37a0b54-1b3b-4899-b7cc-bb6295abd52c-kube-api-access-mlldb\") pod \"network-check-source-7bc5b6bf85-ln2tp\" (UID: \"e37a0b54-1b3b-4899-b7cc-bb6295abd52c\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp" Jul 21 04:09:50.184230 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.184210 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t76tf\" (UniqueName: \"kubernetes.io/projected/564513d6-3759-4aa0-9318-bc34a30a6bbb-kube-api-access-t76tf\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.209722 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.209687 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:50.248586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.248543 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-7r77f" Jul 21 04:09:50.260825 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.260801 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:09:50.260825 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.260819 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:09:50.260981 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.260819 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:09:50.263882 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.263863 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 04:09:50.264119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.264102 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 04:09:50.264225 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.264157 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qwqtb\"" Jul 21 04:09:50.264275 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.264219 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9v8hm\"" Jul 21 04:09:50.276589 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.276570 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" Jul 21 04:09:50.283743 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.283723 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27" Jul 21 04:09:50.291439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.291419 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" Jul 21 04:09:50.298173 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.298153 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:09:50.321110 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.321089 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp" Jul 21 04:09:50.327736 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.327717 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwcbj" Jul 21 04:09:50.666470 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.666428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:50.666648 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.666509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.666648 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.666546 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:50.666648 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.666590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:50.666648 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666608 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666678 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls podName:564513d6-3759-4aa0-9318-bc34a30a6bbb nodeName:}" failed. No retries permitted until 2026-07-21 04:09:51.666662221 +0000 UTC m=+37.065994976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls") pod "dns-default-wx6xf" (UID: "564513d6-3759-4aa0-9318-bc34a30a6bbb") : secret "dns-default-metrics-tls" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666681 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666696 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbfd644f8-thjd2: secret "image-registry-tls" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666699 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:09:51.666688682 +0000 UTC m=+37.066021425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : configmap references non-existent config key: service-ca.crt Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.666722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666738 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls podName:bfda70bf-8536-4acd-a548-fa8edd8b5ba8 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:51.666723736 +0000 UTC m=+37.066056477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls") pod "image-registry-5fbfd644f8-thjd2" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8") : secret "image-registry-tls" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666677 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666787 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert podName:70bac8e0-64e1-437e-878e-d67cf04577ce nodeName:}" failed. No retries permitted until 2026-07-21 04:09:51.666775082 +0000 UTC m=+37.066107821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-6b4vp" (UID: "70bac8e0-64e1-437e-878e-d67cf04577ce") : secret "networking-console-plugin-cert" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666787 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.666783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666823 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls podName:6889f625-36b3-431e-9b82-c71e379c1cee nodeName:}" failed. No retries permitted until 2026-07-21 04:09:51.666812589 +0000 UTC m=+37.066145327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zzs2c" (UID: "6889f625-36b3-431e-9b82-c71e379c1cee") : secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666840 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:50.666857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:50.666903 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666878 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls podName:176aa68a-fa60-43e4-9f48-229ca8a92394 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:51.666865069 +0000 UTC m=+37.066197808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jms6k" (UID: "176aa68a-fa60-43e4-9f48-229ca8a92394") : secret "samples-operator-tls" not found Jul 21 04:09:50.667475 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666922 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 04:09:50.667475 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:50.666961 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:09:51.666951955 +0000 UTC m=+37.066284694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : secret "router-metrics-certs-default" not found Jul 21 04:09:51.618443 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.618263 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6"] Jul 21 04:09:51.627108 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:51.627061 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4297ba8e_5cf6_40a5_90ba_2d1215b9237f.slice/crio-e049870132c7a3b051cf9034bd928d1a86faf23f2159d560c4911fa1ccd2e7e3 WatchSource:0}: Error finding container e049870132c7a3b051cf9034bd928d1a86faf23f2159d560c4911fa1ccd2e7e3: Status 404 returned error can't find the container with id e049870132c7a3b051cf9034bd928d1a86faf23f2159d560c4911fa1ccd2e7e3 Jul 21 04:09:51.635137 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.633487 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-7r77f"] Jul 21 04:09:51.638942 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.638872 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp"] Jul 21 04:09:51.672173 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.672146 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5"] Jul 21 04:09:51.673134 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.673099 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwcbj"] Jul 21 04:09:51.674008 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.673984 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-g9hm8"] Jul 21 04:09:51.674884 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.674859 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27"] Jul 21 04:09:51.676521 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.676277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:51.676521 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.676392 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:51.676521 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.676431 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:51.676521 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.676471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:51.676521 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.676497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:51.676776 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.676536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:51.676776 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:51.676577 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:51.676776 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676688 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 04:09:51.676776 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676718 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 04:09:51.676776 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676753 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert podName:70bac8e0-64e1-437e-878e-d67cf04577ce nodeName:}" failed. No retries permitted until 2026-07-21 04:09:53.676734324 +0000 UTC m=+39.076067079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-6b4vp" (UID: "70bac8e0-64e1-437e-878e-d67cf04577ce") : secret "networking-console-plugin-cert" not found Jul 21 04:09:51.676776 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676753 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 04:09:51.676776 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676769 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbfd644f8-thjd2: secret "image-registry-tls" not found Jul 21 04:09:51.677053 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676780 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls podName:564513d6-3759-4aa0-9318-bc34a30a6bbb nodeName:}" failed. No retries permitted until 2026-07-21 04:09:53.676761468 +0000 UTC m=+39.076094214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls") pod "dns-default-wx6xf" (UID: "564513d6-3759-4aa0-9318-bc34a30a6bbb") : secret "dns-default-metrics-tls" not found Jul 21 04:09:51.677053 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676690 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 04:09:51.677053 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676802 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls podName:bfda70bf-8536-4acd-a548-fa8edd8b5ba8 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:53.676791453 +0000 UTC m=+39.076124203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls") pod "image-registry-5fbfd644f8-thjd2" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8") : secret "image-registry-tls" not found Jul 21 04:09:51.677053 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676803 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:51.677053 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676821 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:09:53.676809802 +0000 UTC m=+39.076142540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : secret "router-metrics-certs-default" not found Jul 21 04:09:51.677053 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676837 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls podName:6889f625-36b3-431e-9b82-c71e379c1cee nodeName:}" failed. No retries permitted until 2026-07-21 04:09:53.676828115 +0000 UTC m=+39.076160880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zzs2c" (UID: "6889f625-36b3-431e-9b82-c71e379c1cee") : secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:51.677053 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676853 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 04:09:51.677053 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676887 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls podName:176aa68a-fa60-43e4-9f48-229ca8a92394 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:53.676876036 +0000 UTC m=+39.076208773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jms6k" (UID: "176aa68a-fa60-43e4-9f48-229ca8a92394") : secret "samples-operator-tls" not found Jul 21 04:09:51.677053 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:51.676962 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:09:53.676944376 +0000 UTC m=+39.076277113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : configmap references non-existent config key: service-ca.crt Jul 21 04:09:51.678796 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:51.678774 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97374277_5b6f_4196_b591_b152148b4b33.slice/crio-e66f0eb5179fef3cad4818dfee6d69e4a0aeb7b1a344f7c854159536cc0e667c WatchSource:0}: Error finding container e66f0eb5179fef3cad4818dfee6d69e4a0aeb7b1a344f7c854159536cc0e667c: Status 404 returned error can't find the container with id e66f0eb5179fef3cad4818dfee6d69e4a0aeb7b1a344f7c854159536cc0e667c Jul 21 04:09:51.679284 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:51.679266 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6317ead8_1226_4ac5_ac8a_89e094b02e72.slice/crio-1fc50dc719a52d24faa45710a311cfa1a8714f28b7d521808ba30cb55e890e29 WatchSource:0}: Error finding container 1fc50dc719a52d24faa45710a311cfa1a8714f28b7d521808ba30cb55e890e29: Status 404 returned error can't find the container with id 1fc50dc719a52d24faa45710a311cfa1a8714f28b7d521808ba30cb55e890e29 Jul 21 04:09:51.680698 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:51.680550 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d55741_8411_4b6b_91fd_599aeb651f83.slice/crio-f0e472ee916d56372138d7853a799e9c3cddf4d04b2470391267099d2e6ede60 WatchSource:0}: Error finding container f0e472ee916d56372138d7853a799e9c3cddf4d04b2470391267099d2e6ede60: Status 404 returned error can't find the container with id f0e472ee916d56372138d7853a799e9c3cddf4d04b2470391267099d2e6ede60 Jul 21 04:09:51.681984 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:51.681867 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8543d9_2530_4536_b660_85e318bf3d60.slice/crio-85d92b483f65562bc91f37297b761b88463e6ca9c4b39f025bdae1ed63d0428d WatchSource:0}: Error finding container 85d92b483f65562bc91f37297b761b88463e6ca9c4b39f025bdae1ed63d0428d: Status 404 returned error can't find the container with id 85d92b483f65562bc91f37297b761b88463e6ca9c4b39f025bdae1ed63d0428d Jul 21 04:09:51.683163 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:51.683060 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37a0b54_1b3b_4899_b7cc_bb6295abd52c.slice/crio-4d1158d61d46b25dcaf5c7c512ce75e135dcaf993054d168f45a903bab4f9a8f WatchSource:0}: Error finding container 4d1158d61d46b25dcaf5c7c512ce75e135dcaf993054d168f45a903bab4f9a8f: Status 404 returned error can't find the container with id 4d1158d61d46b25dcaf5c7c512ce75e135dcaf993054d168f45a903bab4f9a8f Jul 21 04:09:51.683949 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:09:51.683929 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05da316_6261_4ce7_9c1e_53ffa7ff58bc.slice/crio-e86268d8814b3303defa4ca20895dbcdc89a515bcafa87cd431e1d0dcdefa16e WatchSource:0}: Error finding container e86268d8814b3303defa4ca20895dbcdc89a515bcafa87cd431e1d0dcdefa16e: Status 404 returned error can't find the container with id e86268d8814b3303defa4ca20895dbcdc89a515bcafa87cd431e1d0dcdefa16e Jul 21 04:09:52.500424 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:52.500320 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" event={"ID":"6317ead8-1226-4ac5-ac8a-89e094b02e72","Type":"ContainerStarted","Data":"1fc50dc719a52d24faa45710a311cfa1a8714f28b7d521808ba30cb55e890e29"} Jul 21 04:09:52.502677 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:52.502646 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwcbj" event={"ID":"8f8543d9-2530-4536-b660-85e318bf3d60","Type":"ContainerStarted","Data":"85d92b483f65562bc91f37297b761b88463e6ca9c4b39f025bdae1ed63d0428d"} Jul 21 04:09:52.505929 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:52.505902 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27" event={"ID":"d05da316-6261-4ce7-9c1e-53ffa7ff58bc","Type":"ContainerStarted","Data":"e86268d8814b3303defa4ca20895dbcdc89a515bcafa87cd431e1d0dcdefa16e"} Jul 21 04:09:52.508054 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:52.507972 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" event={"ID":"4297ba8e-5cf6-40a5-90ba-2d1215b9237f","Type":"ContainerStarted","Data":"e049870132c7a3b051cf9034bd928d1a86faf23f2159d560c4911fa1ccd2e7e3"} Jul 21 04:09:52.509267 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:52.509194 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-7r77f" event={"ID":"19d55741-8411-4b6b-91fd-599aeb651f83","Type":"ContainerStarted","Data":"f0e472ee916d56372138d7853a799e9c3cddf4d04b2470391267099d2e6ede60"} Jul 21 04:09:52.512454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:52.512384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp" event={"ID":"e37a0b54-1b3b-4899-b7cc-bb6295abd52c","Type":"ContainerStarted","Data":"4d1158d61d46b25dcaf5c7c512ce75e135dcaf993054d168f45a903bab4f9a8f"} Jul 21 04:09:52.514679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:52.514606 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" event={"ID":"97374277-5b6f-4196-b591-b152148b4b33","Type":"ContainerStarted","Data":"e66f0eb5179fef3cad4818dfee6d69e4a0aeb7b1a344f7c854159536cc0e667c"} Jul 21 04:09:52.525154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:52.525047 2567 generic.go:356] "Generic (PLEG): container finished" podID="3e5da763-6747-4693-af09-9370b50f6e56" containerID="06207518828a917159791d72873c9897721e0135d34229f5626e8d29e6db774d" exitCode=0 Jul 21 04:09:52.525154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:52.525081 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" event={"ID":"3e5da763-6747-4693-af09-9370b50f6e56","Type":"ContainerDied","Data":"06207518828a917159791d72873c9897721e0135d34229f5626e8d29e6db774d"} Jul 21 04:09:53.537559 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:53.536348 2567 generic.go:356] "Generic (PLEG): container finished" podID="3e5da763-6747-4693-af09-9370b50f6e56" containerID="0b6788b579b468695d5323f520fa1828377db199667a6879b7270d73e729b2c7" exitCode=0 Jul 21 04:09:53.537559 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:53.536622 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" event={"ID":"3e5da763-6747-4693-af09-9370b50f6e56","Type":"ContainerDied","Data":"0b6788b579b468695d5323f520fa1828377db199667a6879b7270d73e729b2c7"} Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:53.710574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:53.710629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:53.710695 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:53.710803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:53.710840 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:53.710881 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:53.710908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.711025 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.711088 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls podName:6889f625-36b3-431e-9b82-c71e379c1cee nodeName:}" failed. No retries permitted until 2026-07-21 04:09:57.711067074 +0000 UTC m=+43.110399832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zzs2c" (UID: "6889f625-36b3-431e-9b82-c71e379c1cee") : secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.711719 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.711767 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls podName:176aa68a-fa60-43e4-9f48-229ca8a92394 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:57.71175175 +0000 UTC m=+43.111084491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jms6k" (UID: "176aa68a-fa60-43e4-9f48-229ca8a92394") : secret "samples-operator-tls" not found Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.711824 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.711855 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:09:57.711844462 +0000 UTC m=+43.111177213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : secret "router-metrics-certs-default" not found Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.711911 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 04:09:53.712879 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.711939 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls podName:564513d6-3759-4aa0-9318-bc34a30a6bbb nodeName:}" failed. No retries permitted until 2026-07-21 04:09:57.711927914 +0000 UTC m=+43.111260653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls") pod "dns-default-wx6xf" (UID: "564513d6-3759-4aa0-9318-bc34a30a6bbb") : secret "dns-default-metrics-tls" not found Jul 21 04:09:53.713725 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.712320 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:09:57.712307537 +0000 UTC m=+43.111640272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : configmap references non-existent config key: service-ca.crt Jul 21 04:09:53.713725 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.712403 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 04:09:53.713725 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.712440 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert podName:70bac8e0-64e1-437e-878e-d67cf04577ce nodeName:}" failed. No retries permitted until 2026-07-21 04:09:57.71242609 +0000 UTC m=+43.111758829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-6b4vp" (UID: "70bac8e0-64e1-437e-878e-d67cf04577ce") : secret "networking-console-plugin-cert" not found Jul 21 04:09:53.713725 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.712501 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 04:09:53.713725 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.712512 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbfd644f8-thjd2: secret "image-registry-tls" not found Jul 21 04:09:53.713725 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:53.712543 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls podName:bfda70bf-8536-4acd-a548-fa8edd8b5ba8 nodeName:}" failed. No retries permitted until 2026-07-21 04:09:57.712532652 +0000 UTC m=+43.111865395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls") pod "image-registry-5fbfd644f8-thjd2" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8") : secret "image-registry-tls" not found Jul 21 04:09:54.444994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:54.444959 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:54.508813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:54.508732 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:54.546071 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:54.546037 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" event={"ID":"3e5da763-6747-4693-af09-9370b50f6e56","Type":"ContainerStarted","Data":"9554a8c9f14471404d6c8b842c4ac5ac67afeecac8eb5d80e7ce4551cab41771"} Jul 21 04:09:54.571638 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:54.571532 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dtmdb" podStartSLOduration=10.090125345 podStartE2EDuration="39.571514217s" totalImagesPullingTime="29.481388872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:54.569981019 +0000 UTC m=+39.969313808" watchObservedRunningTime="2026-07-21 04:09:54.571514217 +0000 UTC m=+39.970846976" Jul 21 04:09:54.695970 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:54.695927 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:54.754509 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:54.754471 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:55.457530 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:55.457490 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:55.509827 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:55.509796 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:57.495525 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:57.495485 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:57.558317 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:57.558279 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:57.751531 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:57.751451 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:09:57.751531 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:57.751517 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:57.751545 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:57.751570 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:57.751596 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751618 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:57.751633 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:57.751663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751692 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:10:05.751668857 +0000 UTC m=+51.151001606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : configmap references non-existent config key: service-ca.crt Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751698 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751716 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751738 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751752 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls podName:564513d6-3759-4aa0-9318-bc34a30a6bbb nodeName:}" failed. No retries permitted until 2026-07-21 04:10:05.751721142 +0000 UTC m=+51.151053892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls") pod "dns-default-wx6xf" (UID: "564513d6-3759-4aa0-9318-bc34a30a6bbb") : secret "dns-default-metrics-tls" not found Jul 21 04:09:57.751750 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751719 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 04:09:57.752141 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751774 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbfd644f8-thjd2: secret "image-registry-tls" not found Jul 21 04:09:57.752141 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751781 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert podName:70bac8e0-64e1-437e-878e-d67cf04577ce nodeName:}" failed. No retries permitted until 2026-07-21 04:10:05.751770112 +0000 UTC m=+51.151102851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-6b4vp" (UID: "70bac8e0-64e1-437e-878e-d67cf04577ce") : secret "networking-console-plugin-cert" not found Jul 21 04:09:57.752141 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751794 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls podName:176aa68a-fa60-43e4-9f48-229ca8a92394 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:05.751788837 +0000 UTC m=+51.151121573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jms6k" (UID: "176aa68a-fa60-43e4-9f48-229ca8a92394") : secret "samples-operator-tls" not found Jul 21 04:09:57.752141 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751722 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 04:09:57.752141 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751830 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls podName:6889f625-36b3-431e-9b82-c71e379c1cee nodeName:}" failed. No retries permitted until 2026-07-21 04:10:05.751803961 +0000 UTC m=+51.151136701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zzs2c" (UID: "6889f625-36b3-431e-9b82-c71e379c1cee") : secret "cluster-monitoring-operator-tls" not found Jul 21 04:09:57.752141 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751850 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls podName:bfda70bf-8536-4acd-a548-fa8edd8b5ba8 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:05.751842713 +0000 UTC m=+51.151175449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls") pod "image-registry-5fbfd644f8-thjd2" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8") : secret "image-registry-tls" not found Jul 21 04:09:57.752141 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:09:57.751861 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:10:05.751855945 +0000 UTC m=+51.151188681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : secret "router-metrics-certs-default" not found Jul 21 04:09:58.834686 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:58.834654 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:58.934888 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:58.934856 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:58.995757 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:58.995727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:59.418400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.418367 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:59.518759 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.518731 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:59.570592 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.570121 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" event={"ID":"6317ead8-1226-4ac5-ac8a-89e094b02e72","Type":"ContainerStarted","Data":"32edb1ec54bd105bd800d030e59f635b5a8515abc69e0ac3e9a9c2c1902be88c"} Jul 21 04:09:59.573187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.573114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwcbj" event={"ID":"8f8543d9-2530-4536-b660-85e318bf3d60","Type":"ContainerStarted","Data":"3265dca2d00ca93d142900ae8c4a9a14e6ff99d0e1dd823c33997cdc2ea32767"} Jul 21 04:09:59.578613 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.578576 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27" event={"ID":"d05da316-6261-4ce7-9c1e-53ffa7ff58bc","Type":"ContainerStarted","Data":"615b25300b53ecea1b7f7ee8237f6c068b633ff9abcbfcd4be0f4fa783fe55fc"} Jul 21 04:09:59.583147 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.581802 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" event={"ID":"4297ba8e-5cf6-40a5-90ba-2d1215b9237f","Type":"ContainerStarted","Data":"a0eac26a76ab14b36fafbbd8644c2cb19fc3b48d9b8e7b0455a494ba4629f7ef"} Jul 21 04:09:59.583809 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.583783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-7r77f" event={"ID":"19d55741-8411-4b6b-91fd-599aeb651f83","Type":"ContainerStarted","Data":"3b7e9324a8baf42a93de9aa79c2a14ad8d40c9632d6d32e332c1da73f41ae54c"} Jul 21 04:09:59.589265 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.589234 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp" event={"ID":"e37a0b54-1b3b-4899-b7cc-bb6295abd52c","Type":"ContainerStarted","Data":"f18b1798280771af8ad2353ed33cea623473fa6696def80701afc20ad5421833"} Jul 21 04:09:59.591416 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.590984 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" podStartSLOduration=15.984711508 podStartE2EDuration="23.590965839s" totalImagesPullingTime="7.606254331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:59.590295061 +0000 UTC m=+44.989627820" watchObservedRunningTime="2026-07-21 04:09:59.590965839 +0000 UTC m=+44.990298598" Jul 21 04:09:59.591886 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.591865 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/0.log" Jul 21 04:09:59.591967 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.591908 2567 generic.go:356] "Generic (PLEG): container finished" podID="97374277-5b6f-4196-b591-b152148b4b33" containerID="1cc26eb2c3942dd52d0bae8fb9f5eea1e5326663b056eb518f74072cf3ebb224" exitCode=255 Jul 21 04:09:59.591967 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.591943 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" event={"ID":"97374277-5b6f-4196-b591-b152148b4b33","Type":"ContainerDied","Data":"1cc26eb2c3942dd52d0bae8fb9f5eea1e5326663b056eb518f74072cf3ebb224"} Jul 21 04:09:59.592168 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.592149 2567 scope.go:117] "RemoveContainer" containerID="1cc26eb2c3942dd52d0bae8fb9f5eea1e5326663b056eb518f74072cf3ebb224" Jul 21 04:09:59.618819 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.618351 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" podStartSLOduration=15.964190606 podStartE2EDuration="23.616940177s" totalImagesPullingTime="7.652749571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:59.610900863 +0000 UTC m=+45.010233624" watchObservedRunningTime="2026-07-21 04:09:59.616940177 +0000 UTC m=+45.016272917" Jul 21 04:09:59.657060 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.655854 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-7r77f" podStartSLOduration=16.051597242 podStartE2EDuration="23.65583168s" totalImagesPullingTime="7.604234438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:59.635525515 +0000 UTC m=+45.034858273" watchObservedRunningTime="2026-07-21 04:09:59.65583168 +0000 UTC m=+45.055164430" Jul 21 04:09:59.676172 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.676120 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-4nc27" podStartSLOduration=17.701696107 podStartE2EDuration="23.676100673s" totalImagesPullingTime="5.974404566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:59.675591991 +0000 UTC m=+45.074924767" watchObservedRunningTime="2026-07-21 04:09:59.676100673 +0000 UTC m=+45.075433434" Jul 21 04:09:59.676528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.676494 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hwcbj" podStartSLOduration=3.073749609 podStartE2EDuration="10.676479234s" totalImagesPullingTime="7.602729625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:59.658365629 +0000 UTC m=+45.057698387" watchObservedRunningTime="2026-07-21 04:09:59.676479234 +0000 UTC m=+45.075811989" Jul 21 04:09:59.693775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.693705 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-ln2tp" podStartSLOduration=16.082109077 podStartE2EDuration="23.693686211s" totalImagesPullingTime="7.611577134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:09:59.692914307 +0000 UTC m=+45.092247057" watchObservedRunningTime="2026-07-21 04:09:59.693686211 +0000 UTC m=+45.093018972" Jul 21 04:09:59.761127 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.761095 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:09:59.826289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:09:59.826258 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:00.299394 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.299361 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:10:00.299842 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.299430 2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:10:00.597690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.597612 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/1.log" Jul 21 04:10:00.598083 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.598067 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/0.log" Jul 21 04:10:00.598148 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.598102 2567 generic.go:356] "Generic (PLEG): container finished" podID="97374277-5b6f-4196-b591-b152148b4b33" containerID="9cf53131af9767a12a6e600489b7319690f45e4d22096529e5abd93ec95d6643" exitCode=255 Jul 21 04:10:00.598667 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.598469 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" event={"ID":"97374277-5b6f-4196-b591-b152148b4b33","Type":"ContainerDied","Data":"9cf53131af9767a12a6e600489b7319690f45e4d22096529e5abd93ec95d6643"} Jul 21 04:10:00.598667 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.598528 2567 scope.go:117] "RemoveContainer" containerID="1cc26eb2c3942dd52d0bae8fb9f5eea1e5326663b056eb518f74072cf3ebb224" Jul 21 04:10:00.599542 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.599509 2567 scope.go:117] "RemoveContainer" containerID="9cf53131af9767a12a6e600489b7319690f45e4d22096529e5abd93ec95d6643" Jul 21 04:10:00.599686 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:00.599666 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-g9hm8_openshift-console-operator(97374277-5b6f-4196-b591-b152148b4b33)\"" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" podUID="97374277-5b6f-4196-b591-b152148b4b33" Jul 21 04:10:00.879946 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.879860 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lfdqt"] Jul 21 04:10:00.916582 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.916542 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lfdqt"] Jul 21 04:10:00.916731 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.916670 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:00.919024 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.919002 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 04:10:00.919191 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.919167 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t87r9\"" Jul 21 04:10:00.919281 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.919263 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 04:10:00.946205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.946173 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:00.980552 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.980523 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/01a716db-4571-4add-99cb-fb4c3ceff522-crio-socket\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:00.980742 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.980579 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:00.980817 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.980768 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/01a716db-4571-4add-99cb-fb4c3ceff522-data-volume\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:00.980874 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.980832 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgqk\" (UniqueName: \"kubernetes.io/projected/01a716db-4571-4add-99cb-fb4c3ceff522-kube-api-access-kcgqk\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:00.980924 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:00.980869 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.049626 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.049593 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:01.081999 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.081970 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgqk\" (UniqueName: \"kubernetes.io/projected/01a716db-4571-4add-99cb-fb4c3ceff522-kube-api-access-kcgqk\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.081999 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.082000 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.082204 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.082062 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/01a716db-4571-4add-99cb-fb4c3ceff522-crio-socket\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.082204 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.082093 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.082204 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:01.082174 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 04:10:01.082204 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:01.082204 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 04:10:01.082407 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:01.082245 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:01.58221748 +0000 UTC m=+46.981550225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : configmap "kube-rbac-proxy" not found Jul 21 04:10:01.082407 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:01.082261 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:01.582254595 +0000 UTC m=+46.981587334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : secret "insights-runtime-extractor-tls" not found Jul 21 04:10:01.082407 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.082316 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/01a716db-4571-4add-99cb-fb4c3ceff522-crio-socket\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.082407 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.082328 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/01a716db-4571-4add-99cb-fb4c3ceff522-data-volume\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.082590 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.082572 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/01a716db-4571-4add-99cb-fb4c3ceff522-data-volume\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.094592 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.094569 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgqk\" (UniqueName: \"kubernetes.io/projected/01a716db-4571-4add-99cb-fb4c3ceff522-kube-api-access-kcgqk\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.149903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.149831 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:01.464953 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.464880 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:01.519857 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.519825 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:01.587747 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.587711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.587926 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.587819 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:01.587926 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:01.587891 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 04:10:01.587926 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:01.587910 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 04:10:01.588082 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:01.587970 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:02.587949362 +0000 UTC m=+47.987282098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : secret "insights-runtime-extractor-tls" not found Jul 21 04:10:01.588082 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:01.587991 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:02.587981479 +0000 UTC m=+47.987314217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : configmap "kube-rbac-proxy" not found Jul 21 04:10:01.606069 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.606044 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/1.log" Jul 21 04:10:01.606396 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.606379 2567 scope.go:117] "RemoveContainer" containerID="9cf53131af9767a12a6e600489b7319690f45e4d22096529e5abd93ec95d6643" Jul 21 04:10:01.606538 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:01.606520 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-g9hm8_openshift-console-operator(97374277-5b6f-4196-b591-b152148b4b33)\"" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" podUID="97374277-5b6f-4196-b591-b152148b4b33" Jul 21 04:10:01.901187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.901157 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:01.960633 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:01.960605 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:02.167597 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:02.167504 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:02.265673 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:02.265632 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" limit=1024 Jul 21 04:10:02.473834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:02.473762 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bkscj_406658cc-5989-4a5e-8ef8-a27780de41a5/dns-node-resolver/0.log" Jul 21 04:10:02.597922 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:02.597886 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:02.598074 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:02.597988 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:02.598074 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:02.598012 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 04:10:02.598143 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:02.598080 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:04.598059168 +0000 UTC m=+49.997391921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : configmap "kube-rbac-proxy" not found Jul 21 04:10:02.598143 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:02.598127 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 04:10:02.598208 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:02.598182 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:04.598169101 +0000 UTC m=+49.997501848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : secret "insights-runtime-extractor-tls" not found Jul 21 04:10:03.472933 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:03.472906 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-snqhr_5ab7d0e5-db4f-4988-80a9-4edbd26c174c/node-ca/0.log" Jul 21 04:10:04.274103 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:04.274075 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hwcbj_8f8543d9-2530-4536-b660-85e318bf3d60/serve-healthcheck-canary/0.log" Jul 21 04:10:04.616362 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:04.616263 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:04.616487 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:04.616404 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:04.616487 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:04.616456 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 04:10:04.616561 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:04.616524 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:08.616508125 +0000 UTC m=+54.015840883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : secret "insights-runtime-extractor-tls" not found Jul 21 04:10:04.616561 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:04.616538 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 04:10:04.616648 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:04.616608 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:08.61659011 +0000 UTC m=+54.015922869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : configmap "kube-rbac-proxy" not found Jul 21 04:10:05.829196 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:05.829159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:05.829213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829295 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829309 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fbfd644f8-thjd2: secret "image-registry-tls" not found Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:05.829327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829295 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829371 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls podName:bfda70bf-8536-4acd-a548-fa8edd8b5ba8 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:21.829357032 +0000 UTC m=+67.228689768 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls") pod "image-registry-5fbfd644f8-thjd2" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8") : secret "image-registry-tls" not found Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829412 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829457 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert podName:70bac8e0-64e1-437e-878e-d67cf04577ce nodeName:}" failed. No retries permitted until 2026-07-21 04:10:21.829440025 +0000 UTC m=+67.228772770 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-6b4vp" (UID: "70bac8e0-64e1-437e-878e-d67cf04577ce") : secret "networking-console-plugin-cert" not found Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829473 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls podName:6889f625-36b3-431e-9b82-c71e379c1cee nodeName:}" failed. No retries permitted until 2026-07-21 04:10:21.82946756 +0000 UTC m=+67.228800296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-zzs2c" (UID: "6889f625-36b3-431e-9b82-c71e379c1cee") : secret "cluster-monitoring-operator-tls" not found Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:05.829509 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:05.829536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:05.829626 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829606 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 04:10:05.830006 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829651 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls podName:176aa68a-fa60-43e4-9f48-229ca8a92394 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:21.829641706 +0000 UTC m=+67.228974442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jms6k" (UID: "176aa68a-fa60-43e4-9f48-229ca8a92394") : secret "samples-operator-tls" not found Jul 21 04:10:05.830006 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:05.829607 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:10:05.830006 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829663 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 04:10:05.830006 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829694 2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 04:10:05.830006 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829706 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls podName:564513d6-3759-4aa0-9318-bc34a30a6bbb nodeName:}" failed. No retries permitted until 2026-07-21 04:10:21.829694517 +0000 UTC m=+67.229027276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls") pod "dns-default-wx6xf" (UID: "564513d6-3759-4aa0-9318-bc34a30a6bbb") : secret "dns-default-metrics-tls" not found Jul 21 04:10:05.830006 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829728 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:10:21.829716834 +0000 UTC m=+67.229049583 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : secret "router-metrics-certs-default" not found Jul 21 04:10:05.830006 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:05.829759 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:05.830006 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:05.829855 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle podName:ded6b4a0-3715-4e78-8bce-7505e977b0aa nodeName:}" failed. No retries permitted until 2026-07-21 04:10:21.829838561 +0000 UTC m=+67.229171301 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle") pod "router-default-c6fb748b6-ncql2" (UID: "ded6b4a0-3715-4e78-8bce-7505e977b0aa") : configmap references non-existent config key: service-ca.crt Jul 21 04:10:08.652736 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:08.652694 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:08.653216 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:08.652791 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:08.653216 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:08.652852 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 04:10:08.653216 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:08.652882 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 04:10:08.653216 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:08.652916 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:16.652901427 +0000 UTC m=+62.052234163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : secret "insights-runtime-extractor-tls" not found Jul 21 04:10:08.653216 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:08.652931 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm podName:01a716db-4571-4add-99cb-fb4c3ceff522 nodeName:}" failed. No retries permitted until 2026-07-21 04:10:16.652923439 +0000 UTC m=+62.052256175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lfdqt" (UID: "01a716db-4571-4add-99cb-fb4c3ceff522") : configmap "kube-rbac-proxy" not found Jul 21 04:10:10.298976 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:10.298944 2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:10:10.299431 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:10.298992 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:10:10.299431 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:10.299310 2567 scope.go:117] "RemoveContainer" containerID="9cf53131af9767a12a6e600489b7319690f45e4d22096529e5abd93ec95d6643" Jul 21 04:10:10.634261 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:10.634181 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:10:10.634640 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:10.634623 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/1.log" Jul 21 04:10:10.634696 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:10.634657 2567 generic.go:356] "Generic (PLEG): container finished" podID="97374277-5b6f-4196-b591-b152148b4b33" containerID="ef18fa553707197936b0ba17f8ad20b81cf78c7d9b657aed1d99b03d788aa2cb" exitCode=255 Jul 21 04:10:10.634731 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:10.634699 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" event={"ID":"97374277-5b6f-4196-b591-b152148b4b33","Type":"ContainerDied","Data":"ef18fa553707197936b0ba17f8ad20b81cf78c7d9b657aed1d99b03d788aa2cb"} Jul 21 04:10:10.634763 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:10.634743 2567 scope.go:117] "RemoveContainer" containerID="9cf53131af9767a12a6e600489b7319690f45e4d22096529e5abd93ec95d6643" Jul 21 04:10:10.635038 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:10.635018 2567 scope.go:117] "RemoveContainer" containerID="ef18fa553707197936b0ba17f8ad20b81cf78c7d9b657aed1d99b03d788aa2cb" Jul 21 04:10:10.635191 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:10.635171 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-g9hm8_openshift-console-operator(97374277-5b6f-4196-b591-b152148b4b33)\"" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" podUID="97374277-5b6f-4196-b591-b152148b4b33" Jul 21 04:10:11.640054 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:11.640027 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:10:16.498202 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:16.498173 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2ssv" Jul 21 04:10:16.725782 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:16.725749 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:16.725945 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:16.725867 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:16.726276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:16.726255 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/01a716db-4571-4add-99cb-fb4c3ceff522-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:16.728064 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:16.728045 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/01a716db-4571-4add-99cb-fb4c3ceff522-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lfdqt\" (UID: \"01a716db-4571-4add-99cb-fb4c3ceff522\") " pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:16.831037 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:16.831003 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t87r9\"" Jul 21 04:10:16.838905 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:16.838881 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lfdqt" Jul 21 04:10:16.964603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:16.964574 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lfdqt"] Jul 21 04:10:16.966948 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:16.966924 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a716db_4571_4add_99cb_fb4c3ceff522.slice/crio-981068c93f301702be981f31e023960c64b94903e126a5486eae785349b8af49 WatchSource:0}: Error finding container 981068c93f301702be981f31e023960c64b94903e126a5486eae785349b8af49: Status 404 returned error can't find the container with id 981068c93f301702be981f31e023960c64b94903e126a5486eae785349b8af49 Jul 21 04:10:17.662120 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:17.662079 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfdqt" event={"ID":"01a716db-4571-4add-99cb-fb4c3ceff522","Type":"ContainerStarted","Data":"981068c93f301702be981f31e023960c64b94903e126a5486eae785349b8af49"} Jul 21 04:10:20.299323 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.299285 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:10:20.299781 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.299369 2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:10:20.299781 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.299714 2567 scope.go:117] "RemoveContainer" containerID="ef18fa553707197936b0ba17f8ad20b81cf78c7d9b657aed1d99b03d788aa2cb" Jul 21 04:10:20.299941 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:10:20.299915 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-g9hm8_openshift-console-operator(97374277-5b6f-4196-b591-b152148b4b33)\"" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" podUID="97374277-5b6f-4196-b591-b152148b4b33" Jul 21 04:10:20.760847 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.760811 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:10:20.761121 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.760878 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:10:20.763775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.763747 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 04:10:20.764753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.764736 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 04:10:20.773301 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.773278 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fabd6623-e601-4828-8b83-673db7105de6-metrics-certs\") pod \"network-metrics-daemon-8w86n\" (UID: \"fabd6623-e601-4828-8b83-673db7105de6\") " pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:10:20.773420 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.773315 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c21ab5db-d116-4b5f-8f61-df4fe3a0ef89-original-pull-secret\") pod \"global-pull-secret-syncer-wn4dg\" (UID: \"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89\") " pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:10:20.862201 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.862161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:10:20.864938 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.864887 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvwfz\" (UniqueName: \"kubernetes.io/projected/3fff3e92-f836-43fd-8c86-f79e0712cdd1-kube-api-access-mvwfz\") pod \"network-check-target-ft8hx\" (UID: \"3fff3e92-f836-43fd-8c86-f79e0712cdd1\") " pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:10:20.875576 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.875363 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qwqtb\"" Jul 21 04:10:20.879494 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.879471 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn4dg" Jul 21 04:10:20.883566 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.883540 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:10:20.887165 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.887142 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9v8hm\"" Jul 21 04:10:20.895297 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:20.895275 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w86n" Jul 21 04:10:21.801772 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.801737 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8w86n"] Jul 21 04:10:21.805244 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:21.805201 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabd6623_e601_4828_8b83_673db7105de6.slice/crio-ba50da827f15a71a45446f4b2946a4857d4c74c988393a93d6293820a962edc8 WatchSource:0}: Error finding container ba50da827f15a71a45446f4b2946a4857d4c74c988393a93d6293820a962edc8: Status 404 returned error can't find the container with id ba50da827f15a71a45446f4b2946a4857d4c74c988393a93d6293820a962edc8 Jul 21 04:10:21.872431 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.872390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:10:21.872638 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.872445 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:21.872638 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.872474 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:10:21.872638 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.872502 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:10:21.872638 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.872520 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:10:21.872638 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.872543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:10:21.872638 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.872569 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:21.873260 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.873226 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ded6b4a0-3715-4e78-8bce-7505e977b0aa-service-ca-bundle\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:21.875244 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.875186 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/176aa68a-fa60-43e4-9f48-229ca8a92394-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jms6k\" (UID: \"176aa68a-fa60-43e4-9f48-229ca8a92394\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:10:21.875370 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.875275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ded6b4a0-3715-4e78-8bce-7505e977b0aa-metrics-certs\") pod \"router-default-c6fb748b6-ncql2\" (UID: \"ded6b4a0-3715-4e78-8bce-7505e977b0aa\") " pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:21.875527 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.875507 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"image-registry-5fbfd644f8-thjd2\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:10:21.875587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.875550 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/564513d6-3759-4aa0-9318-bc34a30a6bbb-metrics-tls\") pod \"dns-default-wx6xf\" (UID: \"564513d6-3759-4aa0-9318-bc34a30a6bbb\") " pod="openshift-dns/dns-default-wx6xf" Jul 21 04:10:21.875587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.875574 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/70bac8e0-64e1-437e-878e-d67cf04577ce-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6b4vp\" (UID: \"70bac8e0-64e1-437e-878e-d67cf04577ce\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:10:21.875687 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:21.875643 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6889f625-36b3-431e-9b82-c71e379c1cee-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-zzs2c\" (UID: \"6889f625-36b3-431e-9b82-c71e379c1cee\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:10:22.019059 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.019025 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wn4dg"] Jul 21 04:10:22.022007 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.021984 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ft8hx"] Jul 21 04:10:22.023539 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:22.023513 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21ab5db_d116_4b5f_8f61_df4fe3a0ef89.slice/crio-c1ae50c19588b1a2a8588afb34c4d4def721915ed2f8546c51510d6312e8d498 WatchSource:0}: Error finding container c1ae50c19588b1a2a8588afb34c4d4def721915ed2f8546c51510d6312e8d498: Status 404 returned error can't find the container with id c1ae50c19588b1a2a8588afb34c4d4def721915ed2f8546c51510d6312e8d498 Jul 21 04:10:22.024252 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:22.024223 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fff3e92_f836_43fd_8c86_f79e0712cdd1.slice/crio-8f72766da1892dc0b65bad81de7359e0b4591fbeeb6baaea24a78518ab2c727e WatchSource:0}: Error finding container 8f72766da1892dc0b65bad81de7359e0b4591fbeeb6baaea24a78518ab2c727e: Status 404 returned error can't find the container with id 8f72766da1892dc0b65bad81de7359e0b4591fbeeb6baaea24a78518ab2c727e Jul 21 04:10:22.039582 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.039565 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jngx5\"" Jul 21 04:10:22.048047 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.048028 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:10:22.060684 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.060665 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4brc7\"" Jul 21 04:10:22.068245 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.068228 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" Jul 21 04:10:22.070778 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.070757 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-69jxl\"" Jul 21 04:10:22.079200 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.079182 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" Jul 21 04:10:22.109481 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.109425 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dw6n5\"" Jul 21 04:10:22.116666 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.116447 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8q48j\"" Jul 21 04:10:22.116838 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.116819 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" Jul 21 04:10:22.124543 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.124498 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:22.138888 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.138641 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b7grb\"" Jul 21 04:10:22.147366 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.147051 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wx6xf" Jul 21 04:10:22.244809 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.244734 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fbfd644f8-thjd2"] Jul 21 04:10:22.256549 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:22.254594 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfda70bf_8536_4acd_a548_fa8edd8b5ba8.slice/crio-518a26c38af2badc797b3424a61d8166d3a3dc5a548085660db6c2f2b298fa5a WatchSource:0}: Error finding container 518a26c38af2badc797b3424a61d8166d3a3dc5a548085660db6c2f2b298fa5a: Status 404 returned error can't find the container with id 518a26c38af2badc797b3424a61d8166d3a3dc5a548085660db6c2f2b298fa5a Jul 21 04:10:22.299214 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.299184 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c"] Jul 21 04:10:22.304355 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.304311 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k"] Jul 21 04:10:22.361713 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.361645 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp"] Jul 21 04:10:22.364584 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:22.364547 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bac8e0_64e1_437e_878e_d67cf04577ce.slice/crio-ee03bb33385c7e6bedef62f733934e2251b2a998df3677ced90a736eb9ea8bfa WatchSource:0}: Error finding container ee03bb33385c7e6bedef62f733934e2251b2a998df3677ced90a736eb9ea8bfa: Status 404 returned error can't find the container with id ee03bb33385c7e6bedef62f733934e2251b2a998df3677ced90a736eb9ea8bfa Jul 21 04:10:22.382586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.382522 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wx6xf"] Jul 21 04:10:22.386238 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:22.386209 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564513d6_3759_4aa0_9318_bc34a30a6bbb.slice/crio-adb1617f981157d3ed54ee6a1c3801efc44172567683cabf9ee6cbd6b241b065 WatchSource:0}: Error finding container adb1617f981157d3ed54ee6a1c3801efc44172567683cabf9ee6cbd6b241b065: Status 404 returned error can't find the container with id adb1617f981157d3ed54ee6a1c3801efc44172567683cabf9ee6cbd6b241b065 Jul 21 04:10:22.593872 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.593783 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c6fb748b6-ncql2"] Jul 21 04:10:22.598148 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:22.598109 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded6b4a0_3715_4e78_8bce_7505e977b0aa.slice/crio-c88ce13281c8e894364d78239c47d905a9a81aa23386821b2475aa6ad4f9ee78 WatchSource:0}: Error finding container c88ce13281c8e894364d78239c47d905a9a81aa23386821b2475aa6ad4f9ee78: Status 404 returned error can't find the container with id c88ce13281c8e894364d78239c47d905a9a81aa23386821b2475aa6ad4f9ee78 Jul 21 04:10:22.681913 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.681829 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wx6xf" event={"ID":"564513d6-3759-4aa0-9318-bc34a30a6bbb","Type":"ContainerStarted","Data":"adb1617f981157d3ed54ee6a1c3801efc44172567683cabf9ee6cbd6b241b065"} Jul 21 04:10:22.683650 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.683585 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" event={"ID":"70bac8e0-64e1-437e-878e-d67cf04577ce","Type":"ContainerStarted","Data":"ee03bb33385c7e6bedef62f733934e2251b2a998df3677ced90a736eb9ea8bfa"} Jul 21 04:10:22.688171 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.688131 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" event={"ID":"bfda70bf-8536-4acd-a548-fa8edd8b5ba8","Type":"ContainerStarted","Data":"47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a"} Jul 21 04:10:22.688408 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.688346 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" event={"ID":"bfda70bf-8536-4acd-a548-fa8edd8b5ba8","Type":"ContainerStarted","Data":"518a26c38af2badc797b3424a61d8166d3a3dc5a548085660db6c2f2b298fa5a"} Jul 21 04:10:22.688568 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.688533 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:10:22.692627 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.691516 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c6fb748b6-ncql2" event={"ID":"ded6b4a0-3715-4e78-8bce-7505e977b0aa","Type":"ContainerStarted","Data":"c88ce13281c8e894364d78239c47d905a9a81aa23386821b2475aa6ad4f9ee78"} Jul 21 04:10:22.693891 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.693843 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wn4dg" event={"ID":"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89","Type":"ContainerStarted","Data":"c1ae50c19588b1a2a8588afb34c4d4def721915ed2f8546c51510d6312e8d498"} Jul 21 04:10:22.696012 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.695984 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8w86n" event={"ID":"fabd6623-e601-4828-8b83-673db7105de6","Type":"ContainerStarted","Data":"ba50da827f15a71a45446f4b2946a4857d4c74c988393a93d6293820a962edc8"} Jul 21 04:10:22.698532 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.698500 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ft8hx" event={"ID":"3fff3e92-f836-43fd-8c86-f79e0712cdd1","Type":"ContainerStarted","Data":"86f84ba38c4da80e3178c80840875b9fb390e09b715087e6d923e82dc5fe4603"} Jul 21 04:10:22.698637 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.698543 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ft8hx" event={"ID":"3fff3e92-f836-43fd-8c86-f79e0712cdd1","Type":"ContainerStarted","Data":"8f72766da1892dc0b65bad81de7359e0b4591fbeeb6baaea24a78518ab2c727e"} Jul 21 04:10:22.699074 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.699036 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:10:22.700709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.700642 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" event={"ID":"176aa68a-fa60-43e4-9f48-229ca8a92394","Type":"ContainerStarted","Data":"65232efc3a30cae9b5e458758ab38663cf04f51b764e1344daaa80bc2942fdf9"} Jul 21 04:10:22.703379 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.703353 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" event={"ID":"6889f625-36b3-431e-9b82-c71e379c1cee","Type":"ContainerStarted","Data":"9dd816e25995e8ad251c83a1d2a44ced067772e603436123de75e6ae4d3a6d05"} Jul 21 04:10:22.705509 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.705486 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfdqt" event={"ID":"01a716db-4571-4add-99cb-fb4c3ceff522","Type":"ContainerStarted","Data":"d56f11d2f2c62f4d61d08532dfe3b75840327c4609c82a1d9a8cf06d9eced5fc"} Jul 21 04:10:22.729823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:22.729739 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" podStartSLOduration=67.729717942 podStartE2EDuration="1m7.729717942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:22.712431616 +0000 UTC m=+68.111764375" watchObservedRunningTime="2026-07-21 04:10:22.729717942 +0000 UTC m=+68.129050702" Jul 21 04:10:23.716871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:23.716394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c6fb748b6-ncql2" event={"ID":"ded6b4a0-3715-4e78-8bce-7505e977b0aa","Type":"ContainerStarted","Data":"9f5d9284042e72107937b1ea93a0565c1693acfde77ecf6a9e648e44542a0e61"} Jul 21 04:10:23.741668 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:23.741081 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ft8hx" podStartSLOduration=68.741061979 podStartE2EDuration="1m8.741061979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:22.730495475 +0000 UTC m=+68.129828233" watchObservedRunningTime="2026-07-21 04:10:23.741061979 +0000 UTC m=+69.140394737" Jul 21 04:10:24.125326 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.125236 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:24.129050 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.128775 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:24.151627 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.151061 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-c6fb748b6-ncql2" podStartSLOduration=48.151036186 podStartE2EDuration="48.151036186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:23.741724336 +0000 UTC m=+69.141057087" watchObservedRunningTime="2026-07-21 04:10:24.151036186 +0000 UTC m=+69.550368949" Jul 21 04:10:24.391563 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.391473 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs"] Jul 21 04:10:24.421301 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.421246 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs"] Jul 21 04:10:24.421482 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.421408 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.424541 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.424164 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 04:10:24.424541 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.424371 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 21 04:10:24.424541 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.424376 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 04:10:24.424800 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.424503 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 04:10:24.501067 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.501033 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36919b04-4893-4095-aaba-9ec6d52c3632-tmp\") pod \"klusterlet-addon-workmgr-6d497bc9d8-hhlgs\" (UID: \"36919b04-4893-4095-aaba-9ec6d52c3632\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.501240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.501152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5dht\" (UniqueName: \"kubernetes.io/projected/36919b04-4893-4095-aaba-9ec6d52c3632-kube-api-access-t5dht\") pod \"klusterlet-addon-workmgr-6d497bc9d8-hhlgs\" (UID: \"36919b04-4893-4095-aaba-9ec6d52c3632\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.501240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.501187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/36919b04-4893-4095-aaba-9ec6d52c3632-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d497bc9d8-hhlgs\" (UID: \"36919b04-4893-4095-aaba-9ec6d52c3632\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.601808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.601767 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5dht\" (UniqueName: \"kubernetes.io/projected/36919b04-4893-4095-aaba-9ec6d52c3632-kube-api-access-t5dht\") pod \"klusterlet-addon-workmgr-6d497bc9d8-hhlgs\" (UID: \"36919b04-4893-4095-aaba-9ec6d52c3632\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.601981 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.601822 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/36919b04-4893-4095-aaba-9ec6d52c3632-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d497bc9d8-hhlgs\" (UID: \"36919b04-4893-4095-aaba-9ec6d52c3632\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.601981 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.601886 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36919b04-4893-4095-aaba-9ec6d52c3632-tmp\") pod \"klusterlet-addon-workmgr-6d497bc9d8-hhlgs\" (UID: \"36919b04-4893-4095-aaba-9ec6d52c3632\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.602301 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.602276 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36919b04-4893-4095-aaba-9ec6d52c3632-tmp\") pod \"klusterlet-addon-workmgr-6d497bc9d8-hhlgs\" (UID: \"36919b04-4893-4095-aaba-9ec6d52c3632\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.605923 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.605896 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/36919b04-4893-4095-aaba-9ec6d52c3632-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d497bc9d8-hhlgs\" (UID: \"36919b04-4893-4095-aaba-9ec6d52c3632\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.614989 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.614939 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5dht\" (UniqueName: \"kubernetes.io/projected/36919b04-4893-4095-aaba-9ec6d52c3632-kube-api-access-t5dht\") pod \"klusterlet-addon-workmgr-6d497bc9d8-hhlgs\" (UID: \"36919b04-4893-4095-aaba-9ec6d52c3632\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.722095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.721953 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8w86n" event={"ID":"fabd6623-e601-4828-8b83-673db7105de6","Type":"ContainerStarted","Data":"31b1c8b917a7b6a5cf5cb8a4c266a9878a78be2d82990e3468cfbc09c6b3ee68"} Jul 21 04:10:24.722095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.722014 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8w86n" event={"ID":"fabd6623-e601-4828-8b83-673db7105de6","Type":"ContainerStarted","Data":"612bffd926aba2924900a3be32950f11526733353225c45c05b3b19ea78cca83"} Jul 21 04:10:24.722627 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.722276 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:24.723702 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.723680 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c6fb748b6-ncql2" Jul 21 04:10:24.734690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.734671 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:24.739097 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:24.738904 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8w86n" podStartSLOduration=67.967373578 podStartE2EDuration="1m9.738887901s" totalImagesPullingTime="1.771514323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:24.738541504 +0000 UTC m=+70.137874263" watchObservedRunningTime="2026-07-21 04:10:24.738887901 +0000 UTC m=+70.138220658" Jul 21 04:10:31.053252 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.053172 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs"] Jul 21 04:10:31.089115 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:31.089082 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36919b04_4893_4095_aaba_9ec6d52c3632.slice/crio-bd2ab33d12f93bd05dbc9a7d9efe2ec9dc1e06c8086bb629c4817b4a69ef7f20 WatchSource:0}: Error finding container bd2ab33d12f93bd05dbc9a7d9efe2ec9dc1e06c8086bb629c4817b4a69ef7f20: Status 404 returned error can't find the container with id bd2ab33d12f93bd05dbc9a7d9efe2ec9dc1e06c8086bb629c4817b4a69ef7f20 Jul 21 04:10:31.749440 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.749325 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfdqt" event={"ID":"01a716db-4571-4add-99cb-fb4c3ceff522","Type":"ContainerStarted","Data":"7aa082f3e2909e35ac117e221a1b9afb34aec91d0e10e16b489364de9a24d39e"} Jul 21 04:10:31.750850 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.750818 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wn4dg" event={"ID":"c21ab5db-d116-4b5f-8f61-df4fe3a0ef89","Type":"ContainerStarted","Data":"e9700cd354a41f85606bf9e13217f04aa6928d01ffb15ddd2d65723c5d3eb6d1"} Jul 21 04:10:31.752587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.752554 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wx6xf" event={"ID":"564513d6-3759-4aa0-9318-bc34a30a6bbb","Type":"ContainerStarted","Data":"f8cad4c47f3a9da4da7ae98174ac28a3c3db02e3a4ce3ac2103c29edacad12b2"} Jul 21 04:10:31.752587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.752591 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wx6xf" event={"ID":"564513d6-3759-4aa0-9318-bc34a30a6bbb","Type":"ContainerStarted","Data":"6e05243d7b4579a58befa0f2f2746cd068083ed419439782c953d1083c0b1026"} Jul 21 04:10:31.752794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.752663 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wx6xf" Jul 21 04:10:31.753812 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.753786 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" event={"ID":"36919b04-4893-4095-aaba-9ec6d52c3632","Type":"ContainerStarted","Data":"bd2ab33d12f93bd05dbc9a7d9efe2ec9dc1e06c8086bb629c4817b4a69ef7f20"} Jul 21 04:10:31.755393 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.755370 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" event={"ID":"70bac8e0-64e1-437e-878e-d67cf04577ce","Type":"ContainerStarted","Data":"cf93addf664e5308d8dc94d1ee34388bb82d7132d1763845e117ef22b29091e4"} Jul 21 04:10:31.757148 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.757129 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" event={"ID":"176aa68a-fa60-43e4-9f48-229ca8a92394","Type":"ContainerStarted","Data":"455181c3043e0b2d18c70159e141f2d71f7d3edcabb46b1255df8cf2693f67d9"} Jul 21 04:10:31.757244 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.757167 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" event={"ID":"176aa68a-fa60-43e4-9f48-229ca8a92394","Type":"ContainerStarted","Data":"a74d4c560fba5135d7733e1d318376a5ccf06a74d92194bca0349ebc9e4db3e8"} Jul 21 04:10:31.758523 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.758502 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" event={"ID":"6889f625-36b3-431e-9b82-c71e379c1cee","Type":"ContainerStarted","Data":"a4f217962161b6570ae25c9af836aabf260314b6b2a145dc43f1932a7d31dc5e"} Jul 21 04:10:31.766414 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.766376 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wn4dg" podStartSLOduration=67.703134616 podStartE2EDuration="1m16.766364736s" totalImagesPullingTime="9.06323012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:31.766301313 +0000 UTC m=+77.165634073" watchObservedRunningTime="2026-07-21 04:10:31.766364736 +0000 UTC m=+77.165697484" Jul 21 04:10:31.784016 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.783968 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-zzs2c" podStartSLOduration=47.010863278 podStartE2EDuration="55.783955577s" totalImagesPullingTime="8.773092299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:31.782904153 +0000 UTC m=+77.182236917" watchObservedRunningTime="2026-07-21 04:10:31.783955577 +0000 UTC m=+77.183288335" Jul 21 04:10:31.801437 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.801392 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jms6k" podStartSLOduration=47.10140812 podStartE2EDuration="55.801375598s" totalImagesPullingTime="8.699967478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:31.801237034 +0000 UTC m=+77.200569792" watchObservedRunningTime="2026-07-21 04:10:31.801375598 +0000 UTC m=+77.200708358" Jul 21 04:10:31.820346 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.820307 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6b4vp" podStartSLOduration=38.099178913 podStartE2EDuration="46.820294812s" totalImagesPullingTime="8.721115899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:31.819431404 +0000 UTC m=+77.218764162" watchObservedRunningTime="2026-07-21 04:10:31.820294812 +0000 UTC m=+77.219627570" Jul 21 04:10:31.838027 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:31.837977 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wx6xf" podStartSLOduration=34.140093272 podStartE2EDuration="42.837957752s" totalImagesPullingTime="8.69786448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:31.837283043 +0000 UTC m=+77.236615802" watchObservedRunningTime="2026-07-21 04:10:31.837957752 +0000 UTC m=+77.237290511" Jul 21 04:10:33.261150 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:33.261118 2567 scope.go:117] "RemoveContainer" containerID="ef18fa553707197936b0ba17f8ad20b81cf78c7d9b657aed1d99b03d788aa2cb" Jul 21 04:10:36.791718 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.791681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lfdqt" event={"ID":"01a716db-4571-4add-99cb-fb4c3ceff522","Type":"ContainerStarted","Data":"6227cf034a8828969812e531b86c30a26900c3f7ee96f04a6d03461b216744d9"} Jul 21 04:10:36.793526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.793493 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" event={"ID":"36919b04-4893-4095-aaba-9ec6d52c3632","Type":"ContainerStarted","Data":"5546d540169b98f489705cdc3272f58047bb3b35ca925f14d013d4851b04373c"} Jul 21 04:10:36.793648 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.793545 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:36.795070 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.795049 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" Jul 21 04:10:36.795520 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.795504 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:10:36.795606 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.795542 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" event={"ID":"97374277-5b6f-4196-b591-b152148b4b33","Type":"ContainerStarted","Data":"190ee39ec7f4165cb708501caae6b7ad815c251003b554b86e2a3b5f8dfa7b60"} Jul 21 04:10:36.795799 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.795780 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:10:36.813643 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.813597 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lfdqt" podStartSLOduration=17.273468236 podStartE2EDuration="36.813583418s" totalImagesPullingTime="19.540115182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:10:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:36.812702961 +0000 UTC m=+82.212035719" watchObservedRunningTime="2026-07-21 04:10:36.813583418 +0000 UTC m=+82.212916176" Jul 21 04:10:36.829465 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.829428 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d497bc9d8-hhlgs" podStartSLOduration=7.257622298 podStartE2EDuration="12.82941572s" totalImagesPullingTime="5.571793422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:36.828426081 +0000 UTC m=+82.227758839" watchObservedRunningTime="2026-07-21 04:10:36.82941572 +0000 UTC m=+82.228748478" Jul 21 04:10:36.851289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:36.851240 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" podStartSLOduration=53.245221361 podStartE2EDuration="1m0.851227055s" totalImagesPullingTime="7.606005694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:36.850200997 +0000 UTC m=+82.249533748" watchObservedRunningTime="2026-07-21 04:10:36.851227055 +0000 UTC m=+82.250559813" Jul 21 04:10:37.193824 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:37.193788 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-g9hm8" Jul 21 04:10:41.766979 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:41.766947 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wx6xf" Jul 21 04:10:42.052384 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:42.052270 2567 patch_prober.go:28] interesting pod/image-registry-5fbfd644f8-thjd2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 21 04:10:42.052384 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:42.052326 2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" podUID="bfda70bf-8536-4acd-a548-fa8edd8b5ba8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 04:10:43.720861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:43.720834 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:10:44.166453 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.166417 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-whl8l"] Jul 21 04:10:44.169310 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.169293 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.171995 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.171939 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 04:10:44.172128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.172086 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zwp2w\"" Jul 21 04:10:44.172128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.172108 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 04:10:44.172255 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.172237 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 04:10:44.173459 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.173440 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 04:10:44.252109 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.252081 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.252262 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.252113 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9ldz\" (UniqueName: \"kubernetes.io/projected/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-kube-api-access-l9ldz\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.252262 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.252135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-textfile\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.252262 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.252150 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-metrics-client-ca\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.252262 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.252175 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-sys\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.252262 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.252242 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-wtmp\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.252459 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.252295 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-root\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.252459 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.252311 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-tls\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.252459 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.252354 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.352710 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.352678 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-root\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.352710 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.352712 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-tls\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353020 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.352738 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353020 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.352774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-root\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353020 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.352793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353020 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.352818 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9ldz\" (UniqueName: \"kubernetes.io/projected/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-kube-api-access-l9ldz\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353020 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.352840 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-textfile\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353020 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.352864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-metrics-client-ca\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353309 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.353161 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-sys\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353309 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.353215 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-wtmp\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353437 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.353413 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-wtmp\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353488 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.353444 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-textfile\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353488 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.353462 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-sys\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353488 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.353476 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.353612 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.353527 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-metrics-client-ca\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.355157 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.355126 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-tls\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.355260 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.355195 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.364411 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.364390 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9ldz\" (UniqueName: \"kubernetes.io/projected/6b2a90b8-0f3b-4be6-93b3-b6de043464e5-kube-api-access-l9ldz\") pod \"node-exporter-whl8l\" (UID: \"6b2a90b8-0f3b-4be6-93b3-b6de043464e5\") " pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.478971 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.478897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-whl8l" Jul 21 04:10:44.489507 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:44.489479 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2a90b8_0f3b_4be6_93b3_b6de043464e5.slice/crio-0d9365212d55788334bae162279c9b878fd7f9e16234e5d33242d51ebdf5e6a3 WatchSource:0}: Error finding container 0d9365212d55788334bae162279c9b878fd7f9e16234e5d33242d51ebdf5e6a3: Status 404 returned error can't find the container with id 0d9365212d55788334bae162279c9b878fd7f9e16234e5d33242d51ebdf5e6a3 Jul 21 04:10:44.828117 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:44.828086 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-whl8l" event={"ID":"6b2a90b8-0f3b-4be6-93b3-b6de043464e5","Type":"ContainerStarted","Data":"0d9365212d55788334bae162279c9b878fd7f9e16234e5d33242d51ebdf5e6a3"} Jul 21 04:10:45.833299 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:45.833259 2567 generic.go:356] "Generic (PLEG): container finished" podID="6b2a90b8-0f3b-4be6-93b3-b6de043464e5" containerID="d839af0d43ce53fe368cf60d760f061106e7dbd67355597ffd7d440cf4566b40" exitCode=0 Jul 21 04:10:45.833774 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:45.833303 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-whl8l" event={"ID":"6b2a90b8-0f3b-4be6-93b3-b6de043464e5","Type":"ContainerDied","Data":"d839af0d43ce53fe368cf60d760f061106e7dbd67355597ffd7d440cf4566b40"} Jul 21 04:10:46.839265 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:46.839224 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-whl8l" event={"ID":"6b2a90b8-0f3b-4be6-93b3-b6de043464e5","Type":"ContainerStarted","Data":"012c2a0d365b53bb782051a9f4b994645b78387bcd264f552caf9f09d54e34c7"} Jul 21 04:10:46.839643 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:46.839284 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-whl8l" event={"ID":"6b2a90b8-0f3b-4be6-93b3-b6de043464e5","Type":"ContainerStarted","Data":"765a4cdf81c46285f996b2fad5c201e0dace5efbaf892327c24b81cf75e1021a"} Jul 21 04:10:46.862370 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:46.862294 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-whl8l" podStartSLOduration=1.8178768810000001 podStartE2EDuration="2.862273721s" totalImagesPullingTime="978.873188ms" totalInitContainerRuntime="65.523652ms" isStatefulPod=true podCreationTimestamp="2026-07-21 04:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:46.859502977 +0000 UTC m=+92.258835734" watchObservedRunningTime="2026-07-21 04:10:46.862273721 +0000 UTC m=+92.261606481" Jul 21 04:10:48.895632 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:48.895598 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fbfd644f8-thjd2"] Jul 21 04:10:54.082543 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.082501 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-659b4db9b9-kvzdr"] Jul 21 04:10:54.223284 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.223250 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-659b4db9b9-kvzdr"] Jul 21 04:10:54.223448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.223379 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.226226 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.226197 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 04:10:54.226365 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.226260 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 04:10:54.226365 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.226358 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l6m6x\"" Jul 21 04:10:54.226495 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.226367 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 04:10:54.226495 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.226392 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 04:10:54.227269 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.227252 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 04:10:54.227269 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.227266 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 04:10:54.227406 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.227381 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 04:10:54.231323 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.231302 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 04:10:54.324633 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.324601 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-oauth-serving-cert\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.324776 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.324648 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-config\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.324776 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.324685 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxn7t\" (UniqueName: \"kubernetes.io/projected/cfecf29e-8e0d-4d4a-800b-9867c27328cf-kube-api-access-pxn7t\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.324776 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.324750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-oauth-config\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.324876 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.324777 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-service-ca\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.324876 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.324801 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-serving-cert\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.324876 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.324816 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-trusted-ca-bundle\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.426034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.425956 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-config\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.426034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.425996 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxn7t\" (UniqueName: \"kubernetes.io/projected/cfecf29e-8e0d-4d4a-800b-9867c27328cf-kube-api-access-pxn7t\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.426228 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.426046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-oauth-config\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.426228 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.426066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-service-ca\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.426228 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.426084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-serving-cert\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.426228 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.426106 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-trusted-ca-bundle\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.426228 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.426136 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-oauth-serving-cert\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.426875 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.426818 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-config\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.426875 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.426837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-service-ca\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.427077 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.427051 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-oauth-serving-cert\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.427204 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.427183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-trusted-ca-bundle\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.428706 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.428683 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-serving-cert\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.428794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.428716 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-oauth-config\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.435077 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.435059 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxn7t\" (UniqueName: \"kubernetes.io/projected/cfecf29e-8e0d-4d4a-800b-9867c27328cf-kube-api-access-pxn7t\") pod \"console-659b4db9b9-kvzdr\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.532478 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.532446 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:10:54.688270 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.688173 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-659b4db9b9-kvzdr"] Jul 21 04:10:54.691230 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:10:54.691202 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfecf29e_8e0d_4d4a_800b_9867c27328cf.slice/crio-15da7b9f5112c326c72ee3ef10173cf38b4ad1ca91847c5b116a3aa493928903 WatchSource:0}: Error finding container 15da7b9f5112c326c72ee3ef10173cf38b4ad1ca91847c5b116a3aa493928903: Status 404 returned error can't find the container with id 15da7b9f5112c326c72ee3ef10173cf38b4ad1ca91847c5b116a3aa493928903 Jul 21 04:10:54.724978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.724949 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ft8hx" Jul 21 04:10:54.869659 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:54.869628 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-659b4db9b9-kvzdr" event={"ID":"cfecf29e-8e0d-4d4a-800b-9867c27328cf","Type":"ContainerStarted","Data":"15da7b9f5112c326c72ee3ef10173cf38b4ad1ca91847c5b116a3aa493928903"} Jul 21 04:10:58.886677 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:58.886632 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-659b4db9b9-kvzdr" event={"ID":"cfecf29e-8e0d-4d4a-800b-9867c27328cf","Type":"ContainerStarted","Data":"fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274"} Jul 21 04:10:58.910223 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:10:58.910174 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-659b4db9b9-kvzdr" podStartSLOduration=1.580586468 podStartE2EDuration="4.910161959s" totalImagesPullingTime="3.329575491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:10:58.908204151 +0000 UTC m=+104.307536909" watchObservedRunningTime="2026-07-21 04:10:58.910161959 +0000 UTC m=+104.309494714" Jul 21 04:11:04.533506 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:04.533472 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:11:04.533899 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:04.533561 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:11:04.539524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:04.539497 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:11:04.911146 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:04.911060 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:11:07.795946 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.795910 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-574bd4c5bf-lqgl9"] Jul 21 04:11:07.798361 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.798322 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.811493 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.811465 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574bd4c5bf-lqgl9"] Jul 21 04:11:07.835416 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.835389 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-service-ca\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.835543 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.835433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvsv4\" (UniqueName: \"kubernetes.io/projected/05fb46b1-609d-4265-8ed6-63fb213c9ad6-kube-api-access-mvsv4\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.835543 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.835526 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-oauth-config\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.835643 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.835569 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-trusted-ca-bundle\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.835643 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.835589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-oauth-serving-cert\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.835722 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.835668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-serving-cert\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.835722 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.835706 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-config\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.936790 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.936758 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-serving-cert\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.936978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.936799 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-config\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.936978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.936817 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-service-ca\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.936978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.936836 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvsv4\" (UniqueName: \"kubernetes.io/projected/05fb46b1-609d-4265-8ed6-63fb213c9ad6-kube-api-access-mvsv4\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.936978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.936871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-oauth-config\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.936978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.936901 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-trusted-ca-bundle\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.936978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.936919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-oauth-serving-cert\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.937689 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.937658 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-service-ca\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.937689 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.937685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-oauth-serving-cert\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.937877 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.937734 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-config\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.938004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.937980 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-trusted-ca-bundle\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.939261 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.939243 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-oauth-config\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.939403 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.939388 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-serving-cert\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:07.946634 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:07.946615 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvsv4\" (UniqueName: \"kubernetes.io/projected/05fb46b1-609d-4265-8ed6-63fb213c9ad6-kube-api-access-mvsv4\") pod \"console-574bd4c5bf-lqgl9\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:08.108589 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:08.108512 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:08.261068 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:08.261039 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574bd4c5bf-lqgl9"] Jul 21 04:11:08.263628 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:11:08.263601 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fb46b1_609d_4265_8ed6_63fb213c9ad6.slice/crio-69379cb2e8cf73a1df6f21e7a0dbf1d2287a2add16bb03329725fa75eb5be406 WatchSource:0}: Error finding container 69379cb2e8cf73a1df6f21e7a0dbf1d2287a2add16bb03329725fa75eb5be406: Status 404 returned error can't find the container with id 69379cb2e8cf73a1df6f21e7a0dbf1d2287a2add16bb03329725fa75eb5be406 Jul 21 04:11:08.922233 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:08.922195 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574bd4c5bf-lqgl9" event={"ID":"05fb46b1-609d-4265-8ed6-63fb213c9ad6","Type":"ContainerStarted","Data":"22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4"} Jul 21 04:11:08.922691 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:08.922259 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574bd4c5bf-lqgl9" event={"ID":"05fb46b1-609d-4265-8ed6-63fb213c9ad6","Type":"ContainerStarted","Data":"69379cb2e8cf73a1df6f21e7a0dbf1d2287a2add16bb03329725fa75eb5be406"} Jul 21 04:11:08.946349 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:08.946293 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-574bd4c5bf-lqgl9" podStartSLOduration=1.9462778699999999 podStartE2EDuration="1.94627787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:11:08.945112669 +0000 UTC m=+114.344445428" watchObservedRunningTime="2026-07-21 04:11:08.94627787 +0000 UTC m=+114.345610628" Jul 21 04:11:13.916487 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:13.916430 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" podUID="bfda70bf-8536-4acd-a548-fa8edd8b5ba8" containerName="registry" containerID="cri-o://47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a" gracePeriod=30 Jul 21 04:11:14.151462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.151438 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:11:14.297880 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.297846 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-image-registry-private-configuration\") pod \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " Jul 21 04:11:14.298045 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.297894 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") pod \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " Jul 21 04:11:14.298045 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.297993 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-installation-pull-secrets\") pod \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " Jul 21 04:11:14.298121 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.298046 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-trusted-ca\") pod \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " Jul 21 04:11:14.298121 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.298077 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-bound-sa-token\") pod \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " Jul 21 04:11:14.298215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.298126 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-ca-trust-extracted\") pod \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " Jul 21 04:11:14.298215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.298155 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9lz8\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-kube-api-access-s9lz8\") pod \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " Jul 21 04:11:14.298215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.298186 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-certificates\") pod \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\" (UID: \"bfda70bf-8536-4acd-a548-fa8edd8b5ba8\") " Jul 21 04:11:14.298582 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.298522 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bfda70bf-8536-4acd-a548-fa8edd8b5ba8" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:11:14.298851 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.298826 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bfda70bf-8536-4acd-a548-fa8edd8b5ba8" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:11:14.300575 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.300548 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bfda70bf-8536-4acd-a548-fa8edd8b5ba8" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:11:14.300709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.300607 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bfda70bf-8536-4acd-a548-fa8edd8b5ba8" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:11:14.300709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.300653 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bfda70bf-8536-4acd-a548-fa8edd8b5ba8" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:11:14.300709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.300689 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-kube-api-access-s9lz8" (OuterVolumeSpecName: "kube-api-access-s9lz8") pod "bfda70bf-8536-4acd-a548-fa8edd8b5ba8" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8"). InnerVolumeSpecName "kube-api-access-s9lz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:11:14.300890 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.300874 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bfda70bf-8536-4acd-a548-fa8edd8b5ba8" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:11:14.306537 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.306515 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bfda70bf-8536-4acd-a548-fa8edd8b5ba8" (UID: "bfda70bf-8536-4acd-a548-fa8edd8b5ba8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:11:14.399419 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.399393 2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-image-registry-private-configuration\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:14.399419 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.399415 2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-tls\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:14.399569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.399427 2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-installation-pull-secrets\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:14.399569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.399438 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-trusted-ca\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:14.399569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.399447 2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-bound-sa-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:14.399569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.399455 2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-ca-trust-extracted\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:14.399569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.399463 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9lz8\" (UniqueName: \"kubernetes.io/projected/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-kube-api-access-s9lz8\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:14.399569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.399471 2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bfda70bf-8536-4acd-a548-fa8edd8b5ba8-registry-certificates\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:14.945487 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.945451 2567 generic.go:356] "Generic (PLEG): container finished" podID="bfda70bf-8536-4acd-a548-fa8edd8b5ba8" containerID="47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a" exitCode=0 Jul 21 04:11:14.945957 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.945502 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" Jul 21 04:11:14.945957 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.945512 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" event={"ID":"bfda70bf-8536-4acd-a548-fa8edd8b5ba8","Type":"ContainerDied","Data":"47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a"} Jul 21 04:11:14.945957 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.945578 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fbfd644f8-thjd2" event={"ID":"bfda70bf-8536-4acd-a548-fa8edd8b5ba8","Type":"ContainerDied","Data":"518a26c38af2badc797b3424a61d8166d3a3dc5a548085660db6c2f2b298fa5a"} Jul 21 04:11:14.945957 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.945610 2567 scope.go:117] "RemoveContainer" containerID="47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a" Jul 21 04:11:14.955281 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.955266 2567 scope.go:117] "RemoveContainer" containerID="47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a" Jul 21 04:11:14.955634 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:11:14.955606 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a\": container with ID starting with 47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a not found: ID does not exist" containerID="47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a" Jul 21 04:11:14.955690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.955647 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a"} err="failed to get container status \"47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a\": rpc error: code = NotFound desc = could not find container \"47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a\": container with ID starting with 47381fc1fa998bd6c4430bb13cde31a99ad968ff1660e7d48fe8148e3c549b4a not found: ID does not exist" Jul 21 04:11:14.968999 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.968953 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fbfd644f8-thjd2"] Jul 21 04:11:14.974163 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:14.974122 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5fbfd644f8-thjd2"] Jul 21 04:11:15.265399 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:15.265366 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfda70bf-8536-4acd-a548-fa8edd8b5ba8" path="/var/lib/kubelet/pods/bfda70bf-8536-4acd-a548-fa8edd8b5ba8/volumes" Jul 21 04:11:18.109442 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:18.109396 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:18.109442 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:18.109456 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:18.114306 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:18.114285 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:18.963668 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:18.963635 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:11:19.010693 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:19.010663 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-659b4db9b9-kvzdr"] Jul 21 04:11:20.969036 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:20.969004 2567 generic.go:356] "Generic (PLEG): container finished" podID="19d55741-8411-4b6b-91fd-599aeb651f83" containerID="3b7e9324a8baf42a93de9aa79c2a14ad8d40c9632d6d32e332c1da73f41ae54c" exitCode=0 Jul 21 04:11:20.969490 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:20.969079 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-7r77f" event={"ID":"19d55741-8411-4b6b-91fd-599aeb651f83","Type":"ContainerDied","Data":"3b7e9324a8baf42a93de9aa79c2a14ad8d40c9632d6d32e332c1da73f41ae54c"} Jul 21 04:11:20.969490 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:20.969444 2567 scope.go:117] "RemoveContainer" containerID="3b7e9324a8baf42a93de9aa79c2a14ad8d40c9632d6d32e332c1da73f41ae54c" Jul 21 04:11:21.974651 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:21.974621 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-7r77f" event={"ID":"19d55741-8411-4b6b-91fd-599aeb651f83","Type":"ContainerStarted","Data":"bcd8d4839dc159d70266c9ec9a86cf8cfb8cb2f70b53de6351a055606f6cacc9"} Jul 21 04:11:25.990364 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:25.990315 2567 generic.go:356] "Generic (PLEG): container finished" podID="6317ead8-1226-4ac5-ac8a-89e094b02e72" containerID="32edb1ec54bd105bd800d030e59f635b5a8515abc69e0ac3e9a9c2c1902be88c" exitCode=0 Jul 21 04:11:25.990699 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:25.990366 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" event={"ID":"6317ead8-1226-4ac5-ac8a-89e094b02e72","Type":"ContainerDied","Data":"32edb1ec54bd105bd800d030e59f635b5a8515abc69e0ac3e9a9c2c1902be88c"} Jul 21 04:11:25.990699 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:25.990694 2567 scope.go:117] "RemoveContainer" containerID="32edb1ec54bd105bd800d030e59f635b5a8515abc69e0ac3e9a9c2c1902be88c" Jul 21 04:11:26.995853 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:26.995824 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-ccdv5" event={"ID":"6317ead8-1226-4ac5-ac8a-89e094b02e72","Type":"ContainerStarted","Data":"c47d5b8ab0ff04d7254bacd09b6615327664aeca8564bb1fdb2b707c18595e56"} Jul 21 04:11:31.012230 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:31.012191 2567 generic.go:356] "Generic (PLEG): container finished" podID="4297ba8e-5cf6-40a5-90ba-2d1215b9237f" containerID="a0eac26a76ab14b36fafbbd8644c2cb19fc3b48d9b8e7b0455a494ba4629f7ef" exitCode=0 Jul 21 04:11:31.012639 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:31.012266 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" event={"ID":"4297ba8e-5cf6-40a5-90ba-2d1215b9237f","Type":"ContainerDied","Data":"a0eac26a76ab14b36fafbbd8644c2cb19fc3b48d9b8e7b0455a494ba4629f7ef"} Jul 21 04:11:31.012688 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:31.012644 2567 scope.go:117] "RemoveContainer" containerID="a0eac26a76ab14b36fafbbd8644c2cb19fc3b48d9b8e7b0455a494ba4629f7ef" Jul 21 04:11:32.018495 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:32.018458 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-zntm6" event={"ID":"4297ba8e-5cf6-40a5-90ba-2d1215b9237f","Type":"ContainerStarted","Data":"80def84e58e57375141b71ac421830ec02cba52f419a185c936a51a8ee0758bd"} Jul 21 04:11:44.035288 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.035241 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-659b4db9b9-kvzdr" podUID="cfecf29e-8e0d-4d4a-800b-9867c27328cf" containerName="console" containerID="cri-o://fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274" gracePeriod=15 Jul 21 04:11:44.281025 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.281000 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:11:44.340558 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.340468 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-service-ca\") pod \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " Jul 21 04:11:44.340558 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.340533 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-serving-cert\") pod \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " Jul 21 04:11:44.340558 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.340553 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxn7t\" (UniqueName: \"kubernetes.io/projected/cfecf29e-8e0d-4d4a-800b-9867c27328cf-kube-api-access-pxn7t\") pod \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " Jul 21 04:11:44.340837 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.340579 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-trusted-ca-bundle\") pod \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " Jul 21 04:11:44.340837 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.340597 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-oauth-serving-cert\") pod \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " Jul 21 04:11:44.340837 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.340620 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-oauth-config\") pod \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " Jul 21 04:11:44.340837 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.340636 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-config\") pod \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\" (UID: \"cfecf29e-8e0d-4d4a-800b-9867c27328cf\") " Jul 21 04:11:44.341029 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.340892 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-service-ca" (OuterVolumeSpecName: "service-ca") pod "cfecf29e-8e0d-4d4a-800b-9867c27328cf" (UID: "cfecf29e-8e0d-4d4a-800b-9867c27328cf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:11:44.341158 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.341128 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cfecf29e-8e0d-4d4a-800b-9867c27328cf" (UID: "cfecf29e-8e0d-4d4a-800b-9867c27328cf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:11:44.341275 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.341160 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cfecf29e-8e0d-4d4a-800b-9867c27328cf" (UID: "cfecf29e-8e0d-4d4a-800b-9867c27328cf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:11:44.341275 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.341154 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-config" (OuterVolumeSpecName: "console-config") pod "cfecf29e-8e0d-4d4a-800b-9867c27328cf" (UID: "cfecf29e-8e0d-4d4a-800b-9867c27328cf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:11:44.342990 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.342961 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfecf29e-8e0d-4d4a-800b-9867c27328cf-kube-api-access-pxn7t" (OuterVolumeSpecName: "kube-api-access-pxn7t") pod "cfecf29e-8e0d-4d4a-800b-9867c27328cf" (UID: "cfecf29e-8e0d-4d4a-800b-9867c27328cf"). InnerVolumeSpecName "kube-api-access-pxn7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:11:44.343084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.342992 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cfecf29e-8e0d-4d4a-800b-9867c27328cf" (UID: "cfecf29e-8e0d-4d4a-800b-9867c27328cf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:11:44.343084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.343006 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cfecf29e-8e0d-4d4a-800b-9867c27328cf" (UID: "cfecf29e-8e0d-4d4a-800b-9867c27328cf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:11:44.441721 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.441667 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-oauth-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:44.441721 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.441714 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:44.441721 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.441725 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-service-ca\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:44.441721 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.441733 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cfecf29e-8e0d-4d4a-800b-9867c27328cf-console-serving-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:44.441721 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.441743 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxn7t\" (UniqueName: \"kubernetes.io/projected/cfecf29e-8e0d-4d4a-800b-9867c27328cf-kube-api-access-pxn7t\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:44.442029 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.441752 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-trusted-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:44.442029 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:44.441760 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cfecf29e-8e0d-4d4a-800b-9867c27328cf-oauth-serving-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:11:45.067696 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.067663 2567 generic.go:356] "Generic (PLEG): container finished" podID="cfecf29e-8e0d-4d4a-800b-9867c27328cf" containerID="fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274" exitCode=0 Jul 21 04:11:45.068101 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.067706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-659b4db9b9-kvzdr" event={"ID":"cfecf29e-8e0d-4d4a-800b-9867c27328cf","Type":"ContainerDied","Data":"fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274"} Jul 21 04:11:45.068101 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.067713 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-659b4db9b9-kvzdr" Jul 21 04:11:45.068101 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.067748 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-659b4db9b9-kvzdr" event={"ID":"cfecf29e-8e0d-4d4a-800b-9867c27328cf","Type":"ContainerDied","Data":"15da7b9f5112c326c72ee3ef10173cf38b4ad1ca91847c5b116a3aa493928903"} Jul 21 04:11:45.068101 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.067768 2567 scope.go:117] "RemoveContainer" containerID="fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274" Jul 21 04:11:45.077512 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.077488 2567 scope.go:117] "RemoveContainer" containerID="fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274" Jul 21 04:11:45.077775 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:11:45.077747 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274\": container with ID starting with fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274 not found: ID does not exist" containerID="fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274" Jul 21 04:11:45.077857 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.077784 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274"} err="failed to get container status \"fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274\": rpc error: code = NotFound desc = could not find container \"fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274\": container with ID starting with fcbd8c098650671d78f0917a8bb6f6381d86732f4b85a60314cf6ccce9b6a274 not found: ID does not exist" Jul 21 04:11:45.091389 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.091366 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-659b4db9b9-kvzdr"] Jul 21 04:11:45.097779 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.097758 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-659b4db9b9-kvzdr"] Jul 21 04:11:45.265312 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:11:45.265277 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfecf29e-8e0d-4d4a-800b-9867c27328cf" path="/var/lib/kubelet/pods/cfecf29e-8e0d-4d4a-800b-9867c27328cf/volumes" Jul 21 04:12:08.983260 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:08.983227 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b55685479-frshb"] Jul 21 04:12:08.983853 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:08.983761 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfecf29e-8e0d-4d4a-800b-9867c27328cf" containerName="console" Jul 21 04:12:08.983853 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:08.983782 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfecf29e-8e0d-4d4a-800b-9867c27328cf" containerName="console" Jul 21 04:12:08.983853 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:08.983801 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfda70bf-8536-4acd-a548-fa8edd8b5ba8" containerName="registry" Jul 21 04:12:08.983853 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:08.983810 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfda70bf-8536-4acd-a548-fa8edd8b5ba8" containerName="registry" Jul 21 04:12:08.984085 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:08.983887 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfecf29e-8e0d-4d4a-800b-9867c27328cf" containerName="console" Jul 21 04:12:08.984085 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:08.983899 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfda70bf-8536-4acd-a548-fa8edd8b5ba8" containerName="registry" Jul 21 04:12:08.986880 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:08.986859 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:08.997027 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:08.997002 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b55685479-frshb"] Jul 21 04:12:09.034269 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.034238 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-oauth-config\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.034472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.034290 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-service-ca\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.034472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.034402 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-oauth-serving-cert\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.034472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.034436 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-config\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.034472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.034463 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-serving-cert\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.034642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.034485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-trusted-ca-bundle\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.034642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.034518 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxvm4\" (UniqueName: \"kubernetes.io/projected/8919a027-ee74-4eb4-9157-52c71c5e40ce-kube-api-access-cxvm4\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.134886 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.134851 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-oauth-serving-cert\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.134886 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.134890 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-config\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.135127 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.134909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-serving-cert\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.135127 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.134925 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-trusted-ca-bundle\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.135127 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.134944 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxvm4\" (UniqueName: \"kubernetes.io/projected/8919a027-ee74-4eb4-9157-52c71c5e40ce-kube-api-access-cxvm4\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.135127 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.134967 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-oauth-config\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.135127 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.135012 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-service-ca\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.135701 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.135669 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-oauth-serving-cert\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.135847 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.135739 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-config\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.135847 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.135807 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-service-ca\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.135954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.135922 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-trusted-ca-bundle\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.137567 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.137542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-oauth-config\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.137643 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.137590 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-serving-cert\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.147239 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.147217 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxvm4\" (UniqueName: \"kubernetes.io/projected/8919a027-ee74-4eb4-9157-52c71c5e40ce-kube-api-access-cxvm4\") pod \"console-7b55685479-frshb\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.297426 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.297393 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:09.431726 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:09.431696 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b55685479-frshb"] Jul 21 04:12:09.434909 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:12:09.434884 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8919a027_ee74_4eb4_9157_52c71c5e40ce.slice/crio-04d27b1956191601e814006fcbda898c2617a1c80514f9d983cd5226dcbba825 WatchSource:0}: Error finding container 04d27b1956191601e814006fcbda898c2617a1c80514f9d983cd5226dcbba825: Status 404 returned error can't find the container with id 04d27b1956191601e814006fcbda898c2617a1c80514f9d983cd5226dcbba825 Jul 21 04:12:10.162997 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:10.162962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b55685479-frshb" event={"ID":"8919a027-ee74-4eb4-9157-52c71c5e40ce","Type":"ContainerStarted","Data":"fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9"} Jul 21 04:12:10.163398 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:10.163018 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b55685479-frshb" event={"ID":"8919a027-ee74-4eb4-9157-52c71c5e40ce","Type":"ContainerStarted","Data":"04d27b1956191601e814006fcbda898c2617a1c80514f9d983cd5226dcbba825"} Jul 21 04:12:10.182576 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:10.182529 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b55685479-frshb" podStartSLOduration=2.182514971 podStartE2EDuration="2.182514971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:12:10.181415741 +0000 UTC m=+175.580748511" watchObservedRunningTime="2026-07-21 04:12:10.182514971 +0000 UTC m=+175.581847729" Jul 21 04:12:19.298224 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:19.298185 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:19.298678 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:19.298254 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:19.302980 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:19.302949 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:20.202552 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:20.202518 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b55685479-frshb" Jul 21 04:12:20.247419 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:20.247387 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574bd4c5bf-lqgl9"] Jul 21 04:12:45.269683 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.269627 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-574bd4c5bf-lqgl9" podUID="05fb46b1-609d-4265-8ed6-63fb213c9ad6" containerName="console" containerID="cri-o://22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4" gracePeriod=15 Jul 21 04:12:45.502645 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.502621 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:12:45.533435 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533361 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-service-ca\") pod \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " Jul 21 04:12:45.533435 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533400 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-trusted-ca-bundle\") pod \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " Jul 21 04:12:45.533435 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533423 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-serving-cert\") pod \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " Jul 21 04:12:45.533682 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533452 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-oauth-serving-cert\") pod \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " Jul 21 04:12:45.533682 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533500 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-config\") pod \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " Jul 21 04:12:45.533682 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533537 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvsv4\" (UniqueName: \"kubernetes.io/projected/05fb46b1-609d-4265-8ed6-63fb213c9ad6-kube-api-access-mvsv4\") pod \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " Jul 21 04:12:45.533682 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533570 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-oauth-config\") pod \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\" (UID: \"05fb46b1-609d-4265-8ed6-63fb213c9ad6\") " Jul 21 04:12:45.533885 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533851 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "05fb46b1-609d-4265-8ed6-63fb213c9ad6" (UID: "05fb46b1-609d-4265-8ed6-63fb213c9ad6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:12:45.533944 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533910 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-config" (OuterVolumeSpecName: "console-config") pod "05fb46b1-609d-4265-8ed6-63fb213c9ad6" (UID: "05fb46b1-609d-4265-8ed6-63fb213c9ad6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:12:45.533944 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533926 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "05fb46b1-609d-4265-8ed6-63fb213c9ad6" (UID: "05fb46b1-609d-4265-8ed6-63fb213c9ad6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:12:45.534056 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.533941 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-service-ca" (OuterVolumeSpecName: "service-ca") pod "05fb46b1-609d-4265-8ed6-63fb213c9ad6" (UID: "05fb46b1-609d-4265-8ed6-63fb213c9ad6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:12:45.534146 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.534123 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-trusted-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:12:45.534209 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.534154 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-oauth-serving-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:12:45.534209 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.534171 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:12:45.534209 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.534186 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05fb46b1-609d-4265-8ed6-63fb213c9ad6-service-ca\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:12:45.535713 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.535667 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "05fb46b1-609d-4265-8ed6-63fb213c9ad6" (UID: "05fb46b1-609d-4265-8ed6-63fb213c9ad6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:12:45.535911 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.535874 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fb46b1-609d-4265-8ed6-63fb213c9ad6-kube-api-access-mvsv4" (OuterVolumeSpecName: "kube-api-access-mvsv4") pod "05fb46b1-609d-4265-8ed6-63fb213c9ad6" (UID: "05fb46b1-609d-4265-8ed6-63fb213c9ad6"). InnerVolumeSpecName "kube-api-access-mvsv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:12:45.536015 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.535993 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "05fb46b1-609d-4265-8ed6-63fb213c9ad6" (UID: "05fb46b1-609d-4265-8ed6-63fb213c9ad6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:12:45.634755 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.634717 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvsv4\" (UniqueName: \"kubernetes.io/projected/05fb46b1-609d-4265-8ed6-63fb213c9ad6-kube-api-access-mvsv4\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:12:45.634755 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.634749 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-oauth-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:12:45.634755 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:45.634759 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05fb46b1-609d-4265-8ed6-63fb213c9ad6-console-serving-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:12:46.289625 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:46.289583 2567 generic.go:356] "Generic (PLEG): container finished" podID="05fb46b1-609d-4265-8ed6-63fb213c9ad6" containerID="22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4" exitCode=0 Jul 21 04:12:46.289625 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:46.289629 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574bd4c5bf-lqgl9" Jul 21 04:12:46.290188 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:46.289637 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574bd4c5bf-lqgl9" event={"ID":"05fb46b1-609d-4265-8ed6-63fb213c9ad6","Type":"ContainerDied","Data":"22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4"} Jul 21 04:12:46.290188 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:46.289676 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574bd4c5bf-lqgl9" event={"ID":"05fb46b1-609d-4265-8ed6-63fb213c9ad6","Type":"ContainerDied","Data":"69379cb2e8cf73a1df6f21e7a0dbf1d2287a2add16bb03329725fa75eb5be406"} Jul 21 04:12:46.290188 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:46.289696 2567 scope.go:117] "RemoveContainer" containerID="22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4" Jul 21 04:12:46.297968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:46.297950 2567 scope.go:117] "RemoveContainer" containerID="22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4" Jul 21 04:12:46.298211 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:12:46.298191 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4\": container with ID starting with 22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4 not found: ID does not exist" containerID="22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4" Jul 21 04:12:46.298272 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:46.298225 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4"} err="failed to get container status \"22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4\": rpc error: code = NotFound desc = could not find container \"22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4\": container with ID starting with 22bd95e36d7e35f437c093e558466eb21f3721eb5fba86b7d681b166a11a7cf4 not found: ID does not exist" Jul 21 04:12:46.313660 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:46.313634 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574bd4c5bf-lqgl9"] Jul 21 04:12:46.317813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:46.317793 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-574bd4c5bf-lqgl9"] Jul 21 04:12:47.265520 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:12:47.265487 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05fb46b1-609d-4265-8ed6-63fb213c9ad6" path="/var/lib/kubelet/pods/05fb46b1-609d-4265-8ed6-63fb213c9ad6/volumes" Jul 21 04:13:07.093535 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.093495 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn"] Jul 21 04:13:07.093925 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.093770 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05fb46b1-609d-4265-8ed6-63fb213c9ad6" containerName="console" Jul 21 04:13:07.093925 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.093780 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="05fb46b1-609d-4265-8ed6-63fb213c9ad6" containerName="console" Jul 21 04:13:07.093925 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.093840 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="05fb46b1-609d-4265-8ed6-63fb213c9ad6" containerName="console" Jul 21 04:13:07.096563 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.096544 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.099097 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.099073 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 04:13:07.099213 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.099094 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 04:13:07.099213 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.099132 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fjkcz\"" Jul 21 04:13:07.106614 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.106591 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn"] Jul 21 04:13:07.200197 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.200154 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.200197 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.200200 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcbxl\" (UniqueName: \"kubernetes.io/projected/d601ada8-56e3-425f-8351-ab0ece6b6060-kube-api-access-gcbxl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.200441 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.200292 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.301416 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.301378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.301585 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.301461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.301585 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.301481 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcbxl\" (UniqueName: \"kubernetes.io/projected/d601ada8-56e3-425f-8351-ab0ece6b6060-kube-api-access-gcbxl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.301857 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.301832 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.301940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.301853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.312130 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.312101 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcbxl\" (UniqueName: \"kubernetes.io/projected/d601ada8-56e3-425f-8351-ab0ece6b6060-kube-api-access-gcbxl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.406149 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.406062 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:07.558555 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:07.558430 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn"] Jul 21 04:13:07.560885 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:13:07.560845 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd601ada8_56e3_425f_8351_ab0ece6b6060.slice/crio-5d7690a3bcef39ee051d6ca95b4356d53f40be535691a7d2384cb4a4852aed5d WatchSource:0}: Error finding container 5d7690a3bcef39ee051d6ca95b4356d53f40be535691a7d2384cb4a4852aed5d: Status 404 returned error can't find the container with id 5d7690a3bcef39ee051d6ca95b4356d53f40be535691a7d2384cb4a4852aed5d Jul 21 04:13:08.373471 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:08.373429 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" event={"ID":"d601ada8-56e3-425f-8351-ab0ece6b6060","Type":"ContainerStarted","Data":"5d7690a3bcef39ee051d6ca95b4356d53f40be535691a7d2384cb4a4852aed5d"} Jul 21 04:13:14.399317 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:14.399284 2567 generic.go:356] "Generic (PLEG): container finished" podID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerID="3d741a8795219a9787ef42afe5b0b6960e6237d216da77dd0658720d7a9b158a" exitCode=0 Jul 21 04:13:14.399703 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:14.399332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" event={"ID":"d601ada8-56e3-425f-8351-ab0ece6b6060","Type":"ContainerDied","Data":"3d741a8795219a9787ef42afe5b0b6960e6237d216da77dd0658720d7a9b158a"} Jul 21 04:13:16.409219 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:16.409187 2567 generic.go:356] "Generic (PLEG): container finished" podID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerID="0f341af3a605deb77d28062176d51fd89e2906ab493dc1489c58cf5c81027361" exitCode=0 Jul 21 04:13:16.409570 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:16.409229 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" event={"ID":"d601ada8-56e3-425f-8351-ab0ece6b6060","Type":"ContainerDied","Data":"0f341af3a605deb77d28062176d51fd89e2906ab493dc1489c58cf5c81027361"} Jul 21 04:13:23.440549 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:23.440515 2567 generic.go:356] "Generic (PLEG): container finished" podID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerID="2e7749d634c456b0cf80e0fb2e756070a01a1f7398c93a8cb1e5421ea7560f32" exitCode=0 Jul 21 04:13:23.441046 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:23.440601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" event={"ID":"d601ada8-56e3-425f-8351-ab0ece6b6060","Type":"ContainerDied","Data":"2e7749d634c456b0cf80e0fb2e756070a01a1f7398c93a8cb1e5421ea7560f32"} Jul 21 04:13:24.579171 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.579147 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:24.654657 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.654626 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-util\") pod \"d601ada8-56e3-425f-8351-ab0ece6b6060\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " Jul 21 04:13:24.654821 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.654722 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcbxl\" (UniqueName: \"kubernetes.io/projected/d601ada8-56e3-425f-8351-ab0ece6b6060-kube-api-access-gcbxl\") pod \"d601ada8-56e3-425f-8351-ab0ece6b6060\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " Jul 21 04:13:24.654821 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.654750 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-bundle\") pod \"d601ada8-56e3-425f-8351-ab0ece6b6060\" (UID: \"d601ada8-56e3-425f-8351-ab0ece6b6060\") " Jul 21 04:13:24.655446 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.655413 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-bundle" (OuterVolumeSpecName: "bundle") pod "d601ada8-56e3-425f-8351-ab0ece6b6060" (UID: "d601ada8-56e3-425f-8351-ab0ece6b6060"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:13:24.656909 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.656881 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d601ada8-56e3-425f-8351-ab0ece6b6060-kube-api-access-gcbxl" (OuterVolumeSpecName: "kube-api-access-gcbxl") pod "d601ada8-56e3-425f-8351-ab0ece6b6060" (UID: "d601ada8-56e3-425f-8351-ab0ece6b6060"). InnerVolumeSpecName "kube-api-access-gcbxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:13:24.658831 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.658805 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-util" (OuterVolumeSpecName: "util") pod "d601ada8-56e3-425f-8351-ab0ece6b6060" (UID: "d601ada8-56e3-425f-8351-ab0ece6b6060"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:13:24.756086 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.755998 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:13:24.756086 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.756027 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d601ada8-56e3-425f-8351-ab0ece6b6060-util\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:13:24.756086 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:24.756037 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcbxl\" (UniqueName: \"kubernetes.io/projected/d601ada8-56e3-425f-8351-ab0ece6b6060-kube-api-access-gcbxl\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:13:25.450037 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:25.450007 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" event={"ID":"d601ada8-56e3-425f-8351-ab0ece6b6060","Type":"ContainerDied","Data":"5d7690a3bcef39ee051d6ca95b4356d53f40be535691a7d2384cb4a4852aed5d"} Jul 21 04:13:25.450037 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:25.450034 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g4bsn" Jul 21 04:13:25.450227 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:25.450055 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d7690a3bcef39ee051d6ca95b4356d53f40be535691a7d2384cb4a4852aed5d" Jul 21 04:13:30.018862 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.018824 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6"] Jul 21 04:13:30.019264 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.019100 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerName="pull" Jul 21 04:13:30.019264 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.019114 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerName="pull" Jul 21 04:13:30.019264 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.019130 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerName="extract" Jul 21 04:13:30.019264 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.019136 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerName="extract" Jul 21 04:13:30.019264 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.019149 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerName="util" Jul 21 04:13:30.019264 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.019157 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerName="util" Jul 21 04:13:30.019264 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.019203 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d601ada8-56e3-425f-8351-ab0ece6b6060" containerName="extract" Jul 21 04:13:30.058833 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.058800 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6"] Jul 21 04:13:30.058981 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.058911 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" Jul 21 04:13:30.061570 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.061548 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 21 04:13:30.061721 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.061603 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 21 04:13:30.061721 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.061627 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jhrcv\"" Jul 21 04:13:30.201633 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.201604 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/389e849c-d993-454e-91f9-e5116da00b7c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmfj6\" (UID: \"389e849c-d993-454e-91f9-e5116da00b7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" Jul 21 04:13:30.201818 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.201651 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfj7x\" (UniqueName: \"kubernetes.io/projected/389e849c-d993-454e-91f9-e5116da00b7c-kube-api-access-pfj7x\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmfj6\" (UID: \"389e849c-d993-454e-91f9-e5116da00b7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" Jul 21 04:13:30.302634 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.302546 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/389e849c-d993-454e-91f9-e5116da00b7c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmfj6\" (UID: \"389e849c-d993-454e-91f9-e5116da00b7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" Jul 21 04:13:30.302634 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.302604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfj7x\" (UniqueName: \"kubernetes.io/projected/389e849c-d993-454e-91f9-e5116da00b7c-kube-api-access-pfj7x\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmfj6\" (UID: \"389e849c-d993-454e-91f9-e5116da00b7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" Jul 21 04:13:30.302939 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.302921 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/389e849c-d993-454e-91f9-e5116da00b7c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmfj6\" (UID: \"389e849c-d993-454e-91f9-e5116da00b7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" Jul 21 04:13:30.311735 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.311706 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfj7x\" (UniqueName: \"kubernetes.io/projected/389e849c-d993-454e-91f9-e5116da00b7c-kube-api-access-pfj7x\") pod \"cert-manager-operator-controller-manager-55c444bf86-dmfj6\" (UID: \"389e849c-d993-454e-91f9-e5116da00b7c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" Jul 21 04:13:30.367869 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.367838 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" Jul 21 04:13:30.518588 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:30.518563 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6"] Jul 21 04:13:30.520114 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:13:30.520085 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389e849c_d993_454e_91f9_e5116da00b7c.slice/crio-6b88adc8a526a130ba85f7bfeca37d4980496ea517fca4eb2591235c3e0bfaf2 WatchSource:0}: Error finding container 6b88adc8a526a130ba85f7bfeca37d4980496ea517fca4eb2591235c3e0bfaf2: Status 404 returned error can't find the container with id 6b88adc8a526a130ba85f7bfeca37d4980496ea517fca4eb2591235c3e0bfaf2 Jul 21 04:13:31.476694 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:31.476641 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" event={"ID":"389e849c-d993-454e-91f9-e5116da00b7c","Type":"ContainerStarted","Data":"6b88adc8a526a130ba85f7bfeca37d4980496ea517fca4eb2591235c3e0bfaf2"} Jul 21 04:13:32.487126 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:32.487043 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" event={"ID":"389e849c-d993-454e-91f9-e5116da00b7c","Type":"ContainerStarted","Data":"1219cb8aa2e6504fd5fa8c18d6c6092a75a2a432cc5dfd1e9db7f14a06297534"} Jul 21 04:13:32.513915 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:32.513858 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dmfj6" podStartSLOduration=0.898059881 podStartE2EDuration="2.51384217s" totalImagesPullingTime="1.615782289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:13:32.511529602 +0000 UTC m=+257.910862359" watchObservedRunningTime="2026-07-21 04:13:32.51384217 +0000 UTC m=+257.913174929" Jul 21 04:13:33.715531 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.715494 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr"] Jul 21 04:13:33.717950 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.717933 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:33.722857 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.722836 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fjkcz\"" Jul 21 04:13:33.723068 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.723049 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 04:13:33.723895 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.723874 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 04:13:33.729928 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.729906 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr"] Jul 21 04:13:33.832387 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.832357 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:33.832562 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.832393 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6s7g\" (UniqueName: \"kubernetes.io/projected/b957917f-c4ca-49ef-8a0e-429becb44763-kube-api-access-x6s7g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:33.832562 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.832425 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:33.932848 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.932809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:33.932848 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.932851 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6s7g\" (UniqueName: \"kubernetes.io/projected/b957917f-c4ca-49ef-8a0e-429becb44763-kube-api-access-x6s7g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:33.933045 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.932871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:33.933242 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.933211 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:33.933388 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.933222 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:33.942720 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:33.942696 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6s7g\" (UniqueName: \"kubernetes.io/projected/b957917f-c4ca-49ef-8a0e-429becb44763-kube-api-access-x6s7g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:34.027412 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:34.027381 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:34.157958 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:34.156385 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr"] Jul 21 04:13:34.159077 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:13:34.159049 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb957917f_c4ca_49ef_8a0e_429becb44763.slice/crio-01bb5af5b6dcef02657f6cb95979a1f8c6c0a81019e82e012ea6e5fec7d8e9a8 WatchSource:0}: Error finding container 01bb5af5b6dcef02657f6cb95979a1f8c6c0a81019e82e012ea6e5fec7d8e9a8: Status 404 returned error can't find the container with id 01bb5af5b6dcef02657f6cb95979a1f8c6c0a81019e82e012ea6e5fec7d8e9a8 Jul 21 04:13:34.496651 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:34.496566 2567 generic.go:356] "Generic (PLEG): container finished" podID="b957917f-c4ca-49ef-8a0e-429becb44763" containerID="d8cdd56880593e388d9422d8610ba47e4fa90a579a96d72256b4ab5ac0db39cd" exitCode=0 Jul 21 04:13:34.496651 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:34.496627 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" event={"ID":"b957917f-c4ca-49ef-8a0e-429becb44763","Type":"ContainerDied","Data":"d8cdd56880593e388d9422d8610ba47e4fa90a579a96d72256b4ab5ac0db39cd"} Jul 21 04:13:34.496888 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:34.496688 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" event={"ID":"b957917f-c4ca-49ef-8a0e-429becb44763","Type":"ContainerStarted","Data":"01bb5af5b6dcef02657f6cb95979a1f8c6c0a81019e82e012ea6e5fec7d8e9a8"} Jul 21 04:13:36.506557 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:36.506517 2567 generic.go:356] "Generic (PLEG): container finished" podID="b957917f-c4ca-49ef-8a0e-429becb44763" containerID="46d65f8f62bd2fb0cc68801357b49d121d30166246d2664cca6b180109894319" exitCode=0 Jul 21 04:13:36.506930 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:36.506614 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" event={"ID":"b957917f-c4ca-49ef-8a0e-429becb44763","Type":"ContainerDied","Data":"46d65f8f62bd2fb0cc68801357b49d121d30166246d2664cca6b180109894319"} Jul 21 04:13:37.512356 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:37.512237 2567 generic.go:356] "Generic (PLEG): container finished" podID="b957917f-c4ca-49ef-8a0e-429becb44763" containerID="75abb2069b294adb87538def5c12f8e61276981b4e620718eb1b90119f30f457" exitCode=0 Jul 21 04:13:37.512356 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:37.512301 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" event={"ID":"b957917f-c4ca-49ef-8a0e-429becb44763","Type":"ContainerDied","Data":"75abb2069b294adb87538def5c12f8e61276981b4e620718eb1b90119f30f457"} Jul 21 04:13:38.643236 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.643213 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:38.773247 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.773163 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-util\") pod \"b957917f-c4ca-49ef-8a0e-429becb44763\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " Jul 21 04:13:38.773247 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.773213 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6s7g\" (UniqueName: \"kubernetes.io/projected/b957917f-c4ca-49ef-8a0e-429becb44763-kube-api-access-x6s7g\") pod \"b957917f-c4ca-49ef-8a0e-429becb44763\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " Jul 21 04:13:38.773247 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.773241 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-bundle\") pod \"b957917f-c4ca-49ef-8a0e-429becb44763\" (UID: \"b957917f-c4ca-49ef-8a0e-429becb44763\") " Jul 21 04:13:38.773724 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.773691 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-bundle" (OuterVolumeSpecName: "bundle") pod "b957917f-c4ca-49ef-8a0e-429becb44763" (UID: "b957917f-c4ca-49ef-8a0e-429becb44763"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:13:38.775355 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.775319 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b957917f-c4ca-49ef-8a0e-429becb44763-kube-api-access-x6s7g" (OuterVolumeSpecName: "kube-api-access-x6s7g") pod "b957917f-c4ca-49ef-8a0e-429becb44763" (UID: "b957917f-c4ca-49ef-8a0e-429becb44763"). InnerVolumeSpecName "kube-api-access-x6s7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:13:38.780901 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.780861 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-util" (OuterVolumeSpecName: "util") pod "b957917f-c4ca-49ef-8a0e-429becb44763" (UID: "b957917f-c4ca-49ef-8a0e-429becb44763"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:13:38.873867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.873831 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-util\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:13:38.873867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.873862 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6s7g\" (UniqueName: \"kubernetes.io/projected/b957917f-c4ca-49ef-8a0e-429becb44763-kube-api-access-x6s7g\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:13:38.873867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:38.873876 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b957917f-c4ca-49ef-8a0e-429becb44763-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:13:39.523133 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.523051 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" event={"ID":"b957917f-c4ca-49ef-8a0e-429becb44763","Type":"ContainerDied","Data":"01bb5af5b6dcef02657f6cb95979a1f8c6c0a81019e82e012ea6e5fec7d8e9a8"} Jul 21 04:13:39.523133 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.523076 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkg6jr" Jul 21 04:13:39.523133 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.523105 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01bb5af5b6dcef02657f6cb95979a1f8c6c0a81019e82e012ea6e5fec7d8e9a8" Jul 21 04:13:39.647497 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.647461 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz"] Jul 21 04:13:39.647859 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.647734 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b957917f-c4ca-49ef-8a0e-429becb44763" containerName="extract" Jul 21 04:13:39.647859 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.647745 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b957917f-c4ca-49ef-8a0e-429becb44763" containerName="extract" Jul 21 04:13:39.647859 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.647763 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b957917f-c4ca-49ef-8a0e-429becb44763" containerName="pull" Jul 21 04:13:39.647859 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.647771 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b957917f-c4ca-49ef-8a0e-429becb44763" containerName="pull" Jul 21 04:13:39.647859 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.647779 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b957917f-c4ca-49ef-8a0e-429becb44763" containerName="util" Jul 21 04:13:39.647859 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.647785 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b957917f-c4ca-49ef-8a0e-429becb44763" containerName="util" Jul 21 04:13:39.647859 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.647830 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b957917f-c4ca-49ef-8a0e-429becb44763" containerName="extract" Jul 21 04:13:39.649913 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.649892 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" Jul 21 04:13:39.652432 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.652410 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vh7vg\"" Jul 21 04:13:39.652515 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.652439 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 04:13:39.652579 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.652532 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 04:13:39.658871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.658844 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz"] Jul 21 04:13:39.782324 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.782221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/521389bb-ac27-45f3-817d-df1482663444-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hqhpz\" (UID: \"521389bb-ac27-45f3-817d-df1482663444\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" Jul 21 04:13:39.782324 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.782275 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkpg9\" (UniqueName: \"kubernetes.io/projected/521389bb-ac27-45f3-817d-df1482663444-kube-api-access-hkpg9\") pod \"cert-manager-cainjector-559cdd8c87-hqhpz\" (UID: \"521389bb-ac27-45f3-817d-df1482663444\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" Jul 21 04:13:39.883696 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.883655 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/521389bb-ac27-45f3-817d-df1482663444-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hqhpz\" (UID: \"521389bb-ac27-45f3-817d-df1482663444\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" Jul 21 04:13:39.883858 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.883712 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkpg9\" (UniqueName: \"kubernetes.io/projected/521389bb-ac27-45f3-817d-df1482663444-kube-api-access-hkpg9\") pod \"cert-manager-cainjector-559cdd8c87-hqhpz\" (UID: \"521389bb-ac27-45f3-817d-df1482663444\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" Jul 21 04:13:39.893285 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.893249 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkpg9\" (UniqueName: \"kubernetes.io/projected/521389bb-ac27-45f3-817d-df1482663444-kube-api-access-hkpg9\") pod \"cert-manager-cainjector-559cdd8c87-hqhpz\" (UID: \"521389bb-ac27-45f3-817d-df1482663444\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" Jul 21 04:13:39.893285 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.893249 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/521389bb-ac27-45f3-817d-df1482663444-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hqhpz\" (UID: \"521389bb-ac27-45f3-817d-df1482663444\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" Jul 21 04:13:39.976191 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:39.976142 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" Jul 21 04:13:40.101295 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:40.101276 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz"] Jul 21 04:13:40.103512 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:13:40.103482 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod521389bb_ac27_45f3_817d_df1482663444.slice/crio-555163efc25446db37cacdc7cca1bf69e3b9e5facca0dae22f9ba38fbf140b1d WatchSource:0}: Error finding container 555163efc25446db37cacdc7cca1bf69e3b9e5facca0dae22f9ba38fbf140b1d: Status 404 returned error can't find the container with id 555163efc25446db37cacdc7cca1bf69e3b9e5facca0dae22f9ba38fbf140b1d Jul 21 04:13:40.528164 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:40.528132 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" event={"ID":"521389bb-ac27-45f3-817d-df1482663444","Type":"ContainerStarted","Data":"555163efc25446db37cacdc7cca1bf69e3b9e5facca0dae22f9ba38fbf140b1d"} Jul 21 04:13:43.543856 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:43.543815 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" event={"ID":"521389bb-ac27-45f3-817d-df1482663444","Type":"ContainerStarted","Data":"0c1c885f635961cdcabe659d1b802bde81aa48acf66af1de31ea15b4815647f8"} Jul 21 04:13:43.560538 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:43.560484 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hqhpz" podStartSLOduration=2.021401605 podStartE2EDuration="4.560470292s" totalImagesPullingTime="2.539068687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:13:43.559592738 +0000 UTC m=+268.958925498" watchObservedRunningTime="2026-07-21 04:13:43.560470292 +0000 UTC m=+268.959803050" Jul 21 04:13:50.395665 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.395632 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7"] Jul 21 04:13:50.397922 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.397903 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.400528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.400504 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fjkcz\"" Jul 21 04:13:50.400641 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.400504 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 04:13:50.401379 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.401362 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 04:13:50.407137 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.407115 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7"] Jul 21 04:13:50.454848 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.454814 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.455017 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.454867 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7m8\" (UniqueName: \"kubernetes.io/projected/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-kube-api-access-st7m8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.455017 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.454938 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.555839 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.555800 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7m8\" (UniqueName: \"kubernetes.io/projected/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-kube-api-access-st7m8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.555839 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.555845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.556034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.555884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.556213 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.556198 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.556265 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.556244 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.564553 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.564530 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7m8\" (UniqueName: \"kubernetes.io/projected/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-kube-api-access-st7m8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.707297 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.707215 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:50.834638 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:50.834603 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7"] Jul 21 04:13:50.837911 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:13:50.837878 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4d5c42_e3b7_4c42_9901_6bbba9ea67f7.slice/crio-0f6e25a29b78ac31fe3f6feb78980c6290928e074494d8406f4a3dff1176f1da WatchSource:0}: Error finding container 0f6e25a29b78ac31fe3f6feb78980c6290928e074494d8406f4a3dff1176f1da: Status 404 returned error can't find the container with id 0f6e25a29b78ac31fe3f6feb78980c6290928e074494d8406f4a3dff1176f1da Jul 21 04:13:51.578967 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:51.578935 2567 generic.go:356] "Generic (PLEG): container finished" podID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerID="a70203721bd309f29812b142f3800b34d7837fc8e489fb342f35b4dca211cd2c" exitCode=0 Jul 21 04:13:51.579316 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:51.579022 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" event={"ID":"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7","Type":"ContainerDied","Data":"a70203721bd309f29812b142f3800b34d7837fc8e489fb342f35b4dca211cd2c"} Jul 21 04:13:51.579316 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:51.579078 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" event={"ID":"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7","Type":"ContainerStarted","Data":"0f6e25a29b78ac31fe3f6feb78980c6290928e074494d8406f4a3dff1176f1da"} Jul 21 04:13:52.585355 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:52.585313 2567 generic.go:356] "Generic (PLEG): container finished" podID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerID="3b15d3acefc3be73236cb7464e0526100be43b3bc0183cfcc1dde9698bf735bd" exitCode=0 Jul 21 04:13:52.585767 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:52.585404 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" event={"ID":"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7","Type":"ContainerDied","Data":"3b15d3acefc3be73236cb7464e0526100be43b3bc0183cfcc1dde9698bf735bd"} Jul 21 04:13:53.591205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.591168 2567 generic.go:356] "Generic (PLEG): container finished" podID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerID="8a4130235db3d00021452bb699dcac6f30d0379e69f4f2db4ddad6a15975d415" exitCode=0 Jul 21 04:13:53.591594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.591236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" event={"ID":"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7","Type":"ContainerDied","Data":"8a4130235db3d00021452bb699dcac6f30d0379e69f4f2db4ddad6a15975d415"} Jul 21 04:13:53.812094 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.812060 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rmr2h"] Jul 21 04:13:53.833066 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.833028 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rmr2h"] Jul 21 04:13:53.833233 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.833117 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rmr2h" Jul 21 04:13:53.835603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.835575 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-c9925\"" Jul 21 04:13:53.881751 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.881656 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9633b045-2565-451a-af73-ce687dc3b798-bound-sa-token\") pod \"cert-manager-55b4d954d5-rmr2h\" (UID: \"9633b045-2565-451a-af73-ce687dc3b798\") " pod="cert-manager/cert-manager-55b4d954d5-rmr2h" Jul 21 04:13:53.881751 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.881735 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b26vv\" (UniqueName: \"kubernetes.io/projected/9633b045-2565-451a-af73-ce687dc3b798-kube-api-access-b26vv\") pod \"cert-manager-55b4d954d5-rmr2h\" (UID: \"9633b045-2565-451a-af73-ce687dc3b798\") " pod="cert-manager/cert-manager-55b4d954d5-rmr2h" Jul 21 04:13:53.983105 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.983063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9633b045-2565-451a-af73-ce687dc3b798-bound-sa-token\") pod \"cert-manager-55b4d954d5-rmr2h\" (UID: \"9633b045-2565-451a-af73-ce687dc3b798\") " pod="cert-manager/cert-manager-55b4d954d5-rmr2h" Jul 21 04:13:53.983298 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.983125 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b26vv\" (UniqueName: \"kubernetes.io/projected/9633b045-2565-451a-af73-ce687dc3b798-kube-api-access-b26vv\") pod \"cert-manager-55b4d954d5-rmr2h\" (UID: \"9633b045-2565-451a-af73-ce687dc3b798\") " pod="cert-manager/cert-manager-55b4d954d5-rmr2h" Jul 21 04:13:53.992479 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.992446 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9633b045-2565-451a-af73-ce687dc3b798-bound-sa-token\") pod \"cert-manager-55b4d954d5-rmr2h\" (UID: \"9633b045-2565-451a-af73-ce687dc3b798\") " pod="cert-manager/cert-manager-55b4d954d5-rmr2h" Jul 21 04:13:53.992584 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:53.992535 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b26vv\" (UniqueName: \"kubernetes.io/projected/9633b045-2565-451a-af73-ce687dc3b798-kube-api-access-b26vv\") pod \"cert-manager-55b4d954d5-rmr2h\" (UID: \"9633b045-2565-451a-af73-ce687dc3b798\") " pod="cert-manager/cert-manager-55b4d954d5-rmr2h" Jul 21 04:13:54.142456 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.142358 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rmr2h" Jul 21 04:13:54.287102 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.287074 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rmr2h"] Jul 21 04:13:54.289518 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:13:54.289489 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9633b045_2565_451a_af73_ce687dc3b798.slice/crio-db162962dbbe3ac046cdfcdca2c61c82a0bdfb7378adaaddeda9256a73d439e6 WatchSource:0}: Error finding container db162962dbbe3ac046cdfcdca2c61c82a0bdfb7378adaaddeda9256a73d439e6: Status 404 returned error can't find the container with id db162962dbbe3ac046cdfcdca2c61c82a0bdfb7378adaaddeda9256a73d439e6 Jul 21 04:13:54.597681 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.597640 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rmr2h" event={"ID":"9633b045-2565-451a-af73-ce687dc3b798","Type":"ContainerStarted","Data":"31ac7f297db8d755d662299bd80082c9c5a0822ae855aa3118192e7dd59b90fe"} Jul 21 04:13:54.598116 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.597692 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rmr2h" event={"ID":"9633b045-2565-451a-af73-ce687dc3b798","Type":"ContainerStarted","Data":"db162962dbbe3ac046cdfcdca2c61c82a0bdfb7378adaaddeda9256a73d439e6"} Jul 21 04:13:54.616331 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.616281 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rmr2h" podStartSLOduration=1.616264756 podStartE2EDuration="1.616264756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:13:54.613959643 +0000 UTC m=+280.013292435" watchObservedRunningTime="2026-07-21 04:13:54.616264756 +0000 UTC m=+280.015597514" Jul 21 04:13:54.726979 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.726948 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:54.788534 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.788501 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st7m8\" (UniqueName: \"kubernetes.io/projected/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-kube-api-access-st7m8\") pod \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " Jul 21 04:13:54.788696 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.788560 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-bundle\") pod \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " Jul 21 04:13:54.789249 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.789220 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-bundle" (OuterVolumeSpecName: "bundle") pod "ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" (UID: "ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:13:54.790563 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.790540 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-kube-api-access-st7m8" (OuterVolumeSpecName: "kube-api-access-st7m8") pod "ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" (UID: "ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7"). InnerVolumeSpecName "kube-api-access-st7m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:13:54.889455 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.889358 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-util\") pod \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\" (UID: \"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7\") " Jul 21 04:13:54.889605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.889524 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7m8\" (UniqueName: \"kubernetes.io/projected/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-kube-api-access-st7m8\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:13:54.889605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.889537 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:13:54.895161 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.895136 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-util" (OuterVolumeSpecName: "util") pod "ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" (UID: "ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:13:54.990256 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:54.990222 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7-util\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:13:55.603834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:55.603747 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" Jul 21 04:13:55.603834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:55.603775 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af54kb7" event={"ID":"ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7","Type":"ContainerDied","Data":"0f6e25a29b78ac31fe3f6feb78980c6290928e074494d8406f4a3dff1176f1da"} Jul 21 04:13:55.603834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:13:55.603821 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f6e25a29b78ac31fe3f6feb78980c6290928e074494d8406f4a3dff1176f1da" Jul 21 04:14:08.067711 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.067673 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6"] Jul 21 04:14:08.068089 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.067951 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerName="extract" Jul 21 04:14:08.068089 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.067961 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerName="extract" Jul 21 04:14:08.068089 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.067977 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerName="pull" Jul 21 04:14:08.068089 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.067982 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerName="pull" Jul 21 04:14:08.068089 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.067988 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerName="util" Jul 21 04:14:08.068089 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.067994 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerName="util" Jul 21 04:14:08.068089 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.068049 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce4d5c42-e3b7-4c42-9901-6bbba9ea67f7" containerName="extract" Jul 21 04:14:08.070871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.070848 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.073494 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.073473 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 04:14:08.073494 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.073487 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 04:14:08.073859 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.073842 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 04:14:08.073922 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.073891 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sznh7\"" Jul 21 04:14:08.076720 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.076702 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 04:14:08.088409 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.088380 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f12d8d5-ea89-4a60-86d1-3cf819c15488-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.088525 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.088426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr6tr\" (UniqueName: \"kubernetes.io/projected/4f12d8d5-ea89-4a60-86d1-3cf819c15488-kube-api-access-qr6tr\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.088525 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.088486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4f12d8d5-ea89-4a60-86d1-3cf819c15488-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.088644 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.088556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f12d8d5-ea89-4a60-86d1-3cf819c15488-tmp\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.088644 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.088587 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4f12d8d5-ea89-4a60-86d1-3cf819c15488-manifests\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.090008 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.089990 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6"] Jul 21 04:14:08.189577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.189539 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f12d8d5-ea89-4a60-86d1-3cf819c15488-tmp\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.189794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.189584 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4f12d8d5-ea89-4a60-86d1-3cf819c15488-manifests\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.189794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.189617 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f12d8d5-ea89-4a60-86d1-3cf819c15488-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.189794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.189642 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr6tr\" (UniqueName: \"kubernetes.io/projected/4f12d8d5-ea89-4a60-86d1-3cf819c15488-kube-api-access-qr6tr\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.189794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.189674 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4f12d8d5-ea89-4a60-86d1-3cf819c15488-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.190104 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.190074 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4f12d8d5-ea89-4a60-86d1-3cf819c15488-manifests\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.191986 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.191948 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f12d8d5-ea89-4a60-86d1-3cf819c15488-tmp\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.192216 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.192197 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f12d8d5-ea89-4a60-86d1-3cf819c15488-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.192285 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.192197 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4f12d8d5-ea89-4a60-86d1-3cf819c15488-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.203026 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.203007 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr6tr\" (UniqueName: \"kubernetes.io/projected/4f12d8d5-ea89-4a60-86d1-3cf819c15488-kube-api-access-qr6tr\") pod \"opendatahub-operator-controller-manager-7d877cbcc8-vpqz6\" (UID: \"4f12d8d5-ea89-4a60-86d1-3cf819c15488\") " pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.380858 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.380766 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:08.542201 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.542152 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6"] Jul 21 04:14:08.544949 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:14:08.544918 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f12d8d5_ea89_4a60_86d1_3cf819c15488.slice/crio-caa1a1ecf26d9b29a83f3d5e5ccd8c868200f8fad7dcc05a6d904079db416fe3 WatchSource:0}: Error finding container caa1a1ecf26d9b29a83f3d5e5ccd8c868200f8fad7dcc05a6d904079db416fe3: Status 404 returned error can't find the container with id caa1a1ecf26d9b29a83f3d5e5ccd8c868200f8fad7dcc05a6d904079db416fe3 Jul 21 04:14:08.662093 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:08.662007 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" event={"ID":"4f12d8d5-ea89-4a60-86d1-3cf819c15488","Type":"ContainerStarted","Data":"caa1a1ecf26d9b29a83f3d5e5ccd8c868200f8fad7dcc05a6d904079db416fe3"} Jul 21 04:14:11.679415 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:11.679315 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" event={"ID":"4f12d8d5-ea89-4a60-86d1-3cf819c15488","Type":"ContainerStarted","Data":"38905c57c7d84f924cc34015e3687cc91acecbd1cbc36e2e41b6823cdc189a66"} Jul 21 04:14:12.685684 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:12.685646 2567 generic.go:356] "Generic (PLEG): container finished" podID="4f12d8d5-ea89-4a60-86d1-3cf819c15488" containerID="38905c57c7d84f924cc34015e3687cc91acecbd1cbc36e2e41b6823cdc189a66" exitCode=0 Jul 21 04:14:12.686094 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:12.685700 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" event={"ID":"4f12d8d5-ea89-4a60-86d1-3cf819c15488","Type":"ContainerDied","Data":"38905c57c7d84f924cc34015e3687cc91acecbd1cbc36e2e41b6823cdc189a66"} Jul 21 04:14:13.691767 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:13.691729 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" event={"ID":"4f12d8d5-ea89-4a60-86d1-3cf819c15488","Type":"ContainerStarted","Data":"06428a222248768a89fe684f4d44a1799a3b20ebe65c9d134b93a6d8e0f4e3f4"} Jul 21 04:14:13.692218 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:13.691838 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:13.716687 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:13.716632 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" podStartSLOduration=1.467097216 podStartE2EDuration="5.716615804s" totalImagesPullingTime="3.19832598s" totalInitContainerRuntime="1.051192608s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:14:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:14:13.714144785 +0000 UTC m=+299.113477554" watchObservedRunningTime="2026-07-21 04:14:13.716615804 +0000 UTC m=+299.115948562" Jul 21 04:14:14.998467 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:14.998442 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:14:15.002697 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:15.002667 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:14:15.010692 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:15.010667 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 04:14:24.698979 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:24.698946 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d877cbcc8-vpqz6" Jul 21 04:14:27.085454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.085415 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq"] Jul 21 04:14:27.088805 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.088789 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.092125 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.092095 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 04:14:27.097367 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.097325 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5wrmc\"" Jul 21 04:14:27.097781 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.097764 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 04:14:27.113398 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.113373 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq"] Jul 21 04:14:27.149041 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.149007 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bdb37ce-942f-4d36-94e4-b09abc0e200b-tls-certs\") pod \"kube-auth-proxy-757f6cbd6c-82drq\" (UID: \"3bdb37ce-942f-4d36-94e4-b09abc0e200b\") " pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.149041 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.149044 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bdb37ce-942f-4d36-94e4-b09abc0e200b-tmp\") pod \"kube-auth-proxy-757f6cbd6c-82drq\" (UID: \"3bdb37ce-942f-4d36-94e4-b09abc0e200b\") " pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.149234 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.149124 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dw84\" (UniqueName: \"kubernetes.io/projected/3bdb37ce-942f-4d36-94e4-b09abc0e200b-kube-api-access-7dw84\") pod \"kube-auth-proxy-757f6cbd6c-82drq\" (UID: \"3bdb37ce-942f-4d36-94e4-b09abc0e200b\") " pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.249577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.249540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bdb37ce-942f-4d36-94e4-b09abc0e200b-tls-certs\") pod \"kube-auth-proxy-757f6cbd6c-82drq\" (UID: \"3bdb37ce-942f-4d36-94e4-b09abc0e200b\") " pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.249577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.249579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bdb37ce-942f-4d36-94e4-b09abc0e200b-tmp\") pod \"kube-auth-proxy-757f6cbd6c-82drq\" (UID: \"3bdb37ce-942f-4d36-94e4-b09abc0e200b\") " pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.249837 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.249635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dw84\" (UniqueName: \"kubernetes.io/projected/3bdb37ce-942f-4d36-94e4-b09abc0e200b-kube-api-access-7dw84\") pod \"kube-auth-proxy-757f6cbd6c-82drq\" (UID: \"3bdb37ce-942f-4d36-94e4-b09abc0e200b\") " pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.251863 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.251829 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bdb37ce-942f-4d36-94e4-b09abc0e200b-tmp\") pod \"kube-auth-proxy-757f6cbd6c-82drq\" (UID: \"3bdb37ce-942f-4d36-94e4-b09abc0e200b\") " pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.252064 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.252046 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bdb37ce-942f-4d36-94e4-b09abc0e200b-tls-certs\") pod \"kube-auth-proxy-757f6cbd6c-82drq\" (UID: \"3bdb37ce-942f-4d36-94e4-b09abc0e200b\") " pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.261327 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.261301 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dw84\" (UniqueName: \"kubernetes.io/projected/3bdb37ce-942f-4d36-94e4-b09abc0e200b-kube-api-access-7dw84\") pod \"kube-auth-proxy-757f6cbd6c-82drq\" (UID: \"3bdb37ce-942f-4d36-94e4-b09abc0e200b\") " pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.357403 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.357311 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l8l94"] Jul 21 04:14:27.360687 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.360670 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:27.364109 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.364085 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jc4zq\"" Jul 21 04:14:27.364273 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.364256 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 04:14:27.377491 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.377460 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l8l94"] Jul 21 04:14:27.398701 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.398675 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" Jul 21 04:14:27.451635 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.451599 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwsz6\" (UniqueName: \"kubernetes.io/projected/c8a817cb-3511-4266-980f-7e1a5ce1c10b-kube-api-access-bwsz6\") pod \"odh-model-controller-7998c67dc4-l8l94\" (UID: \"c8a817cb-3511-4266-980f-7e1a5ce1c10b\") " pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:27.451780 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.451708 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a817cb-3511-4266-980f-7e1a5ce1c10b-cert\") pod \"odh-model-controller-7998c67dc4-l8l94\" (UID: \"c8a817cb-3511-4266-980f-7e1a5ce1c10b\") " pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:27.536012 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.531310 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq"] Jul 21 04:14:27.537431 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.537410 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 04:14:27.552159 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.552127 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a817cb-3511-4266-980f-7e1a5ce1c10b-cert\") pod \"odh-model-controller-7998c67dc4-l8l94\" (UID: \"c8a817cb-3511-4266-980f-7e1a5ce1c10b\") " pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:27.552273 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.552207 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwsz6\" (UniqueName: \"kubernetes.io/projected/c8a817cb-3511-4266-980f-7e1a5ce1c10b-kube-api-access-bwsz6\") pod \"odh-model-controller-7998c67dc4-l8l94\" (UID: \"c8a817cb-3511-4266-980f-7e1a5ce1c10b\") " pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:27.552273 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:14:27.552264 2567 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 04:14:27.552404 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:14:27.552328 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a817cb-3511-4266-980f-7e1a5ce1c10b-cert podName:c8a817cb-3511-4266-980f-7e1a5ce1c10b nodeName:}" failed. No retries permitted until 2026-07-21 04:14:28.052309412 +0000 UTC m=+313.451642150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c8a817cb-3511-4266-980f-7e1a5ce1c10b-cert") pod "odh-model-controller-7998c67dc4-l8l94" (UID: "c8a817cb-3511-4266-980f-7e1a5ce1c10b") : secret "odh-model-controller-webhook-cert" not found Jul 21 04:14:27.563623 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.563598 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwsz6\" (UniqueName: \"kubernetes.io/projected/c8a817cb-3511-4266-980f-7e1a5ce1c10b-kube-api-access-bwsz6\") pod \"odh-model-controller-7998c67dc4-l8l94\" (UID: \"c8a817cb-3511-4266-980f-7e1a5ce1c10b\") " pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:27.756938 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:27.756847 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" event={"ID":"3bdb37ce-942f-4d36-94e4-b09abc0e200b","Type":"ContainerStarted","Data":"33312d5ff55b347ac316496103548fd189b520dbd3eaa6ec79127f2714779b48"} Jul 21 04:14:28.057985 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.057942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a817cb-3511-4266-980f-7e1a5ce1c10b-cert\") pod \"odh-model-controller-7998c67dc4-l8l94\" (UID: \"c8a817cb-3511-4266-980f-7e1a5ce1c10b\") " pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:28.060886 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.060860 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8a817cb-3511-4266-980f-7e1a5ce1c10b-cert\") pod \"odh-model-controller-7998c67dc4-l8l94\" (UID: \"c8a817cb-3511-4266-980f-7e1a5ce1c10b\") " pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:28.270952 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.270914 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:28.424813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.424787 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj"] Jul 21 04:14:28.426734 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:14:28.426707 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a817cb_3511_4266_980f_7e1a5ce1c10b.slice/crio-98a6dbc83207a9a434e4601a65f58443c0eeb10ba1bf731c4bff68034630934d WatchSource:0}: Error finding container 98a6dbc83207a9a434e4601a65f58443c0eeb10ba1bf731c4bff68034630934d: Status 404 returned error can't find the container with id 98a6dbc83207a9a434e4601a65f58443c0eeb10ba1bf731c4bff68034630934d Jul 21 04:14:28.429094 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.429075 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-l8l94"] Jul 21 04:14:28.429199 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.429185 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.431915 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.431896 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 04:14:28.432017 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.431901 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 04:14:28.432084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.432028 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7qbwr\"" Jul 21 04:14:28.432282 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.432261 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 04:14:28.432415 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.432308 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 04:14:28.442517 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.442493 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj"] Jul 21 04:14:28.562860 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.562827 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-token\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.563041 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.562869 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.563041 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.562994 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-cacerts\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.563041 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.563028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.563204 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.563054 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35562e9-e18d-4301-a5b3-3b77f13777ef-local-certs\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.563204 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.563100 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx8gs\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-kube-api-access-kx8gs\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.563204 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.563168 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.663858 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.663764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-token\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.663858 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.663809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.664095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.663884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-cacerts\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.664095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.663907 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.664095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.664036 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35562e9-e18d-4301-a5b3-3b77f13777ef-local-certs\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.664095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.664076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx8gs\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-kube-api-access-kx8gs\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.664292 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.664118 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.664634 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.664585 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.666998 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.666937 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-cacerts\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.666998 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.666982 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35562e9-e18d-4301-a5b3-3b77f13777ef-local-certs\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.666998 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.666991 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.667187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.667009 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.673847 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.673821 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx8gs\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-kube-api-access-kx8gs\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.674173 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.674153 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-token\") pod \"istiod-openshift-gateway-7574459656-5sdgj\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.744315 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.744276 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:28.763818 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.763779 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" event={"ID":"c8a817cb-3511-4266-980f-7e1a5ce1c10b","Type":"ContainerStarted","Data":"98a6dbc83207a9a434e4601a65f58443c0eeb10ba1bf731c4bff68034630934d"} Jul 21 04:14:28.910300 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:28.910249 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj"] Jul 21 04:14:28.910618 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:14:28.910585 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35562e9_e18d_4301_a5b3_3b77f13777ef.slice/crio-1b79982c036f746a60f27a27791fad86d887a332fc807f257628ea64df48c4e2 WatchSource:0}: Error finding container 1b79982c036f746a60f27a27791fad86d887a332fc807f257628ea64df48c4e2: Status 404 returned error can't find the container with id 1b79982c036f746a60f27a27791fad86d887a332fc807f257628ea64df48c4e2 Jul 21 04:14:29.772458 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:29.772416 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" event={"ID":"c35562e9-e18d-4301-a5b3-3b77f13777ef","Type":"ContainerStarted","Data":"1b79982c036f746a60f27a27791fad86d887a332fc807f257628ea64df48c4e2"} Jul 21 04:14:32.331773 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.331738 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:14:32.332211 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.331819 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:14:32.790065 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.790034 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" event={"ID":"c8a817cb-3511-4266-980f-7e1a5ce1c10b","Type":"ContainerStarted","Data":"61d2f68a77c49d56badd48910b797b34b69f4307bc57009707dbfa1c481244ae"} Jul 21 04:14:32.790312 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.790133 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:14:32.791594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.791570 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" event={"ID":"3bdb37ce-942f-4d36-94e4-b09abc0e200b","Type":"ContainerStarted","Data":"e35e35dcfa390c6bc591e1add7e7fc9a4517a7d8cc93a36497027804a4d92b08"} Jul 21 04:14:32.793083 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.793051 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" event={"ID":"c35562e9-e18d-4301-a5b3-3b77f13777ef","Type":"ContainerStarted","Data":"03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4"} Jul 21 04:14:32.793188 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.793129 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:32.807933 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.807888 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" podStartSLOduration=1.9050465989999998 podStartE2EDuration="5.807875991s" totalImagesPullingTime="3.902829392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:14:32.806142789 +0000 UTC m=+318.205475560" watchObservedRunningTime="2026-07-21 04:14:32.807875991 +0000 UTC m=+318.207208727" Jul 21 04:14:32.822442 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.822384 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-757f6cbd6c-82drq" podStartSLOduration=1.103028293 podStartE2EDuration="5.822364032s" totalImagesPullingTime="4.719335739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:14:32.821696022 +0000 UTC m=+318.221028780" watchObservedRunningTime="2026-07-21 04:14:32.822364032 +0000 UTC m=+318.221696788" Jul 21 04:14:32.851248 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:32.851170 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" podStartSLOduration=1.432897974 podStartE2EDuration="4.851148067s" totalImagesPullingTime="3.418250093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:14:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:14:32.85007778 +0000 UTC m=+318.249410539" watchObservedRunningTime="2026-07-21 04:14:32.851148067 +0000 UTC m=+318.250480826" Jul 21 04:14:33.800308 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:33.800280 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:36.832091 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.832058 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn"] Jul 21 04:14:36.835058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.835040 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:36.837660 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.837641 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6sl4k\"" Jul 21 04:14:36.839797 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.839781 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 21 04:14:36.845674 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.845649 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn"] Jul 21 04:14:36.848919 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.848896 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss"] Jul 21 04:14:36.852057 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.852029 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:14:36.854956 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.854932 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2fn9g\"" Jul 21 04:14:36.855071 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.855025 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 04:14:36.874735 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.874702 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss"] Jul 21 04:14:36.950622 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.950586 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q96k\" (UniqueName: \"kubernetes.io/projected/eba1f514-8403-4696-8c2d-cc2670d4f072-kube-api-access-5q96k\") pod \"llmisvc-controller-manager-777c6bd7d5-jtzss\" (UID: \"eba1f514-8403-4696-8c2d-cc2670d4f072\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:14:36.950622 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.950622 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2xr7\" (UniqueName: \"kubernetes.io/projected/fbf45a96-3933-44e2-9eaa-df1310cf26b1-kube-api-access-q2xr7\") pod \"kserve-controller-manager-d6f4b65b8-cv4fn\" (UID: \"fbf45a96-3933-44e2-9eaa-df1310cf26b1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:36.950832 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.950647 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eba1f514-8403-4696-8c2d-cc2670d4f072-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jtzss\" (UID: \"eba1f514-8403-4696-8c2d-cc2670d4f072\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:14:36.950832 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:36.950732 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbf45a96-3933-44e2-9eaa-df1310cf26b1-cert\") pod \"kserve-controller-manager-d6f4b65b8-cv4fn\" (UID: \"fbf45a96-3933-44e2-9eaa-df1310cf26b1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:37.052147 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.052109 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbf45a96-3933-44e2-9eaa-df1310cf26b1-cert\") pod \"kserve-controller-manager-d6f4b65b8-cv4fn\" (UID: \"fbf45a96-3933-44e2-9eaa-df1310cf26b1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:37.052320 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.052194 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q96k\" (UniqueName: \"kubernetes.io/projected/eba1f514-8403-4696-8c2d-cc2670d4f072-kube-api-access-5q96k\") pod \"llmisvc-controller-manager-777c6bd7d5-jtzss\" (UID: \"eba1f514-8403-4696-8c2d-cc2670d4f072\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:14:37.052320 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:14:37.052272 2567 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 21 04:14:37.052447 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:14:37.052363 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbf45a96-3933-44e2-9eaa-df1310cf26b1-cert podName:fbf45a96-3933-44e2-9eaa-df1310cf26b1 nodeName:}" failed. No retries permitted until 2026-07-21 04:14:37.552320359 +0000 UTC m=+322.951653094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fbf45a96-3933-44e2-9eaa-df1310cf26b1-cert") pod "kserve-controller-manager-d6f4b65b8-cv4fn" (UID: "fbf45a96-3933-44e2-9eaa-df1310cf26b1") : secret "kserve-webhook-server-cert" not found Jul 21 04:14:37.052447 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.052388 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2xr7\" (UniqueName: \"kubernetes.io/projected/fbf45a96-3933-44e2-9eaa-df1310cf26b1-kube-api-access-q2xr7\") pod \"kserve-controller-manager-d6f4b65b8-cv4fn\" (UID: \"fbf45a96-3933-44e2-9eaa-df1310cf26b1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:37.052447 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.052427 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eba1f514-8403-4696-8c2d-cc2670d4f072-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jtzss\" (UID: \"eba1f514-8403-4696-8c2d-cc2670d4f072\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:14:37.054910 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.054879 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eba1f514-8403-4696-8c2d-cc2670d4f072-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-jtzss\" (UID: \"eba1f514-8403-4696-8c2d-cc2670d4f072\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:14:37.066649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.066619 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q96k\" (UniqueName: \"kubernetes.io/projected/eba1f514-8403-4696-8c2d-cc2670d4f072-kube-api-access-5q96k\") pod \"llmisvc-controller-manager-777c6bd7d5-jtzss\" (UID: \"eba1f514-8403-4696-8c2d-cc2670d4f072\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:14:37.066764 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.066748 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2xr7\" (UniqueName: \"kubernetes.io/projected/fbf45a96-3933-44e2-9eaa-df1310cf26b1-kube-api-access-q2xr7\") pod \"kserve-controller-manager-d6f4b65b8-cv4fn\" (UID: \"fbf45a96-3933-44e2-9eaa-df1310cf26b1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:37.164174 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.164081 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:14:37.319594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.319570 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss"] Jul 21 04:14:37.321848 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:14:37.321804 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeba1f514_8403_4696_8c2d_cc2670d4f072.slice/crio-5d5466aa340ca9c3f243ad1e188e06c1ff31ccc0969cfb5dcefb0f48e3708025 WatchSource:0}: Error finding container 5d5466aa340ca9c3f243ad1e188e06c1ff31ccc0969cfb5dcefb0f48e3708025: Status 404 returned error can't find the container with id 5d5466aa340ca9c3f243ad1e188e06c1ff31ccc0969cfb5dcefb0f48e3708025 Jul 21 04:14:37.556877 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.556834 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbf45a96-3933-44e2-9eaa-df1310cf26b1-cert\") pod \"kserve-controller-manager-d6f4b65b8-cv4fn\" (UID: \"fbf45a96-3933-44e2-9eaa-df1310cf26b1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:37.559263 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.559240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbf45a96-3933-44e2-9eaa-df1310cf26b1-cert\") pod \"kserve-controller-manager-d6f4b65b8-cv4fn\" (UID: \"fbf45a96-3933-44e2-9eaa-df1310cf26b1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:37.747737 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.747704 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:37.818605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.818537 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" event={"ID":"eba1f514-8403-4696-8c2d-cc2670d4f072","Type":"ContainerStarted","Data":"5d5466aa340ca9c3f243ad1e188e06c1ff31ccc0969cfb5dcefb0f48e3708025"} Jul 21 04:14:37.875466 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:37.875441 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn"] Jul 21 04:14:37.878156 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:14:37.878129 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf45a96_3933_44e2_9eaa_df1310cf26b1.slice/crio-c2845b474994759853c085de0103bceabf849d638c8013c4f5e4418e91d2bc45 WatchSource:0}: Error finding container c2845b474994759853c085de0103bceabf849d638c8013c4f5e4418e91d2bc45: Status 404 returned error can't find the container with id c2845b474994759853c085de0103bceabf849d638c8013c4f5e4418e91d2bc45 Jul 21 04:14:38.827595 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:38.827551 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" event={"ID":"fbf45a96-3933-44e2-9eaa-df1310cf26b1","Type":"ContainerStarted","Data":"c2845b474994759853c085de0103bceabf849d638c8013c4f5e4418e91d2bc45"} Jul 21 04:14:39.875188 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:39.875142 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj"] Jul 21 04:14:39.875740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:39.875404 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" podUID="c35562e9-e18d-4301-a5b3-3b77f13777ef" containerName="discovery" containerID="cri-o://03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4" gracePeriod=30 Jul 21 04:14:40.594678 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.594654 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:40.687955 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.687934 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-cacerts\") pod \"c35562e9-e18d-4301-a5b3-3b77f13777ef\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " Jul 21 04:14:40.688072 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.687974 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-dns-cert\") pod \"c35562e9-e18d-4301-a5b3-3b77f13777ef\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " Jul 21 04:14:40.688072 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.688018 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx8gs\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-kube-api-access-kx8gs\") pod \"c35562e9-e18d-4301-a5b3-3b77f13777ef\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " Jul 21 04:14:40.688072 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.688047 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-kubeconfig\") pod \"c35562e9-e18d-4301-a5b3-3b77f13777ef\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " Jul 21 04:14:40.688236 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.688077 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-token\") pod \"c35562e9-e18d-4301-a5b3-3b77f13777ef\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " Jul 21 04:14:40.688236 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.688102 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-ca-configmap\") pod \"c35562e9-e18d-4301-a5b3-3b77f13777ef\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " Jul 21 04:14:40.688359 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.688235 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35562e9-e18d-4301-a5b3-3b77f13777ef-local-certs\") pod \"c35562e9-e18d-4301-a5b3-3b77f13777ef\" (UID: \"c35562e9-e18d-4301-a5b3-3b77f13777ef\") " Jul 21 04:14:40.688483 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.688452 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c35562e9-e18d-4301-a5b3-3b77f13777ef" (UID: "c35562e9-e18d-4301-a5b3-3b77f13777ef"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:14:40.688619 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.688598 2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-ca-configmap\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:14:40.690449 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.690428 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-cacerts" (OuterVolumeSpecName: "cacerts") pod "c35562e9-e18d-4301-a5b3-3b77f13777ef" (UID: "c35562e9-e18d-4301-a5b3-3b77f13777ef"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:14:40.690647 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.690582 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-kube-api-access-kx8gs" (OuterVolumeSpecName: "kube-api-access-kx8gs") pod "c35562e9-e18d-4301-a5b3-3b77f13777ef" (UID: "c35562e9-e18d-4301-a5b3-3b77f13777ef"). InnerVolumeSpecName "kube-api-access-kx8gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:14:40.690647 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.690613 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c35562e9-e18d-4301-a5b3-3b77f13777ef" (UID: "c35562e9-e18d-4301-a5b3-3b77f13777ef"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:14:40.690647 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.690617 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "c35562e9-e18d-4301-a5b3-3b77f13777ef" (UID: "c35562e9-e18d-4301-a5b3-3b77f13777ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:14:40.690647 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.690633 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c35562e9-e18d-4301-a5b3-3b77f13777ef" (UID: "c35562e9-e18d-4301-a5b3-3b77f13777ef"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:14:40.690809 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.690788 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c35562e9-e18d-4301-a5b3-3b77f13777ef-local-certs" (OuterVolumeSpecName: "local-certs") pod "c35562e9-e18d-4301-a5b3-3b77f13777ef" (UID: "c35562e9-e18d-4301-a5b3-3b77f13777ef"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:14:40.789011 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.788981 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx8gs\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-kube-api-access-kx8gs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:14:40.789011 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.789006 2567 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-kubeconfig\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:14:40.789011 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.789015 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:14:40.789206 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.789023 2567 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c35562e9-e18d-4301-a5b3-3b77f13777ef-local-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:14:40.789206 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.789031 2567 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-cacerts\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:14:40.789206 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.789038 2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c35562e9-e18d-4301-a5b3-3b77f13777ef-istio-csr-dns-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:14:40.839381 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.839252 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" event={"ID":"fbf45a96-3933-44e2-9eaa-df1310cf26b1","Type":"ContainerStarted","Data":"952c1bf83ee1f896638e979cda4faf77648c24691c253af3740422e004593a37"} Jul 21 04:14:40.839534 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.839432 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:14:40.840687 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.840662 2567 generic.go:356] "Generic (PLEG): container finished" podID="c35562e9-e18d-4301-a5b3-3b77f13777ef" containerID="03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4" exitCode=0 Jul 21 04:14:40.840819 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.840696 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" Jul 21 04:14:40.840819 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.840741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" event={"ID":"c35562e9-e18d-4301-a5b3-3b77f13777ef","Type":"ContainerDied","Data":"03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4"} Jul 21 04:14:40.840819 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.840777 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj" event={"ID":"c35562e9-e18d-4301-a5b3-3b77f13777ef","Type":"ContainerDied","Data":"1b79982c036f746a60f27a27791fad86d887a332fc807f257628ea64df48c4e2"} Jul 21 04:14:40.840819 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.840800 2567 scope.go:117] "RemoveContainer" containerID="03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4" Jul 21 04:14:40.842425 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.842402 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" event={"ID":"eba1f514-8403-4696-8c2d-cc2670d4f072","Type":"ContainerStarted","Data":"01bc9a5c151d000b91064ea392fffcb8b621dc4739bd73a49db458c0d9d70ea5"} Jul 21 04:14:40.842544 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.842529 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:14:40.850424 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.850406 2567 scope.go:117] "RemoveContainer" containerID="03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4" Jul 21 04:14:40.850693 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:14:40.850665 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4\": container with ID starting with 03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4 not found: ID does not exist" containerID="03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4" Jul 21 04:14:40.850753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.850692 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4"} err="failed to get container status \"03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4\": rpc error: code = NotFound desc = could not find container \"03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4\": container with ID starting with 03349a5c6686c646ecbe1fc020feb54d168b65b6ba31aab34828fa45040076a4 not found: ID does not exist" Jul 21 04:14:40.858048 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.858006 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" podStartSLOduration=2.112608317 podStartE2EDuration="4.857993811s" totalImagesPullingTime="2.745385494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:14:40.856558406 +0000 UTC m=+326.255891163" watchObservedRunningTime="2026-07-21 04:14:40.857993811 +0000 UTC m=+326.257326577" Jul 21 04:14:40.873300 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.873264 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj"] Jul 21 04:14:40.879450 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.879420 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5sdgj"] Jul 21 04:14:40.900266 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:40.900208 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" podStartSLOduration=1.603836601 podStartE2EDuration="4.900192963s" totalImagesPullingTime="3.296356362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:14:40.896179408 +0000 UTC m=+326.295512167" watchObservedRunningTime="2026-07-21 04:14:40.900192963 +0000 UTC m=+326.299525747" Jul 21 04:14:41.266160 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:41.266125 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c35562e9-e18d-4301-a5b3-3b77f13777ef" path="/var/lib/kubelet/pods/c35562e9-e18d-4301-a5b3-3b77f13777ef/volumes" Jul 21 04:14:43.800398 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:14:43.800332 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-l8l94" Jul 21 04:15:11.850610 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:11.850579 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-jtzss" Jul 21 04:15:11.853551 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:11.853528 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cv4fn" Jul 21 04:15:28.249912 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.249868 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7dp58"] Jul 21 04:15:28.250387 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.250200 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35562e9-e18d-4301-a5b3-3b77f13777ef" containerName="discovery" Jul 21 04:15:28.250387 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.250212 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35562e9-e18d-4301-a5b3-3b77f13777ef" containerName="discovery" Jul 21 04:15:28.250387 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.250290 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35562e9-e18d-4301-a5b3-3b77f13777ef" containerName="discovery" Jul 21 04:15:28.255320 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.255295 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" Jul 21 04:15:28.257903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.257881 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 04:15:28.258049 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.257947 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-xkrp4\"" Jul 21 04:15:28.258864 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.258835 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 04:15:28.267661 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.267633 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7dp58"] Jul 21 04:15:28.379170 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.379137 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrgpr\" (UniqueName: \"kubernetes.io/projected/5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1-kube-api-access-wrgpr\") pod \"kuadrant-operator-catalog-7dp58\" (UID: \"5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1\") " pod="kuadrant-system/kuadrant-operator-catalog-7dp58" Jul 21 04:15:28.480013 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.479968 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrgpr\" (UniqueName: \"kubernetes.io/projected/5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1-kube-api-access-wrgpr\") pod \"kuadrant-operator-catalog-7dp58\" (UID: \"5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1\") " pod="kuadrant-system/kuadrant-operator-catalog-7dp58" Jul 21 04:15:28.505167 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.505093 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrgpr\" (UniqueName: \"kubernetes.io/projected/5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1-kube-api-access-wrgpr\") pod \"kuadrant-operator-catalog-7dp58\" (UID: \"5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1\") " pod="kuadrant-system/kuadrant-operator-catalog-7dp58" Jul 21 04:15:28.568237 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.568195 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" Jul 21 04:15:28.620538 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.620508 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7dp58"] Jul 21 04:15:28.699978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.699952 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7dp58"] Jul 21 04:15:28.702761 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:15:28.702729 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1284a2_f5ae_4384_a1a2_0aa3d0d221b1.slice/crio-2d5dbec166f08f9382bcc7cd907289b139de4fdb7d5baff4568e3ff955789208 WatchSource:0}: Error finding container 2d5dbec166f08f9382bcc7cd907289b139de4fdb7d5baff4568e3ff955789208: Status 404 returned error can't find the container with id 2d5dbec166f08f9382bcc7cd907289b139de4fdb7d5baff4568e3ff955789208 Jul 21 04:15:28.828117 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.828084 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-576mf"] Jul 21 04:15:28.830813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.830795 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-576mf" Jul 21 04:15:28.842487 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.842465 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-576mf"] Jul 21 04:15:28.985428 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:28.985392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkspw\" (UniqueName: \"kubernetes.io/projected/0c9ba075-91dd-440e-ba17-c235f89b3e9f-kube-api-access-lkspw\") pod \"kuadrant-operator-catalog-576mf\" (UID: \"0c9ba075-91dd-440e-ba17-c235f89b3e9f\") " pod="kuadrant-system/kuadrant-operator-catalog-576mf" Jul 21 04:15:29.070478 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:29.070439 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" event={"ID":"5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1","Type":"ContainerStarted","Data":"2d5dbec166f08f9382bcc7cd907289b139de4fdb7d5baff4568e3ff955789208"} Jul 21 04:15:29.086821 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:29.086754 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkspw\" (UniqueName: \"kubernetes.io/projected/0c9ba075-91dd-440e-ba17-c235f89b3e9f-kube-api-access-lkspw\") pod \"kuadrant-operator-catalog-576mf\" (UID: \"0c9ba075-91dd-440e-ba17-c235f89b3e9f\") " pod="kuadrant-system/kuadrant-operator-catalog-576mf" Jul 21 04:15:29.095679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:29.095655 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkspw\" (UniqueName: \"kubernetes.io/projected/0c9ba075-91dd-440e-ba17-c235f89b3e9f-kube-api-access-lkspw\") pod \"kuadrant-operator-catalog-576mf\" (UID: \"0c9ba075-91dd-440e-ba17-c235f89b3e9f\") " pod="kuadrant-system/kuadrant-operator-catalog-576mf" Jul 21 04:15:29.139598 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:29.139571 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-576mf" Jul 21 04:15:29.275724 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:29.275701 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-576mf"] Jul 21 04:15:29.277502 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:15:29.277473 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9ba075_91dd_440e_ba17_c235f89b3e9f.slice/crio-8412394f26d59469866df6ded7a8bb9582ec3aeaa16a12d81b58d7f18c099e36 WatchSource:0}: Error finding container 8412394f26d59469866df6ded7a8bb9582ec3aeaa16a12d81b58d7f18c099e36: Status 404 returned error can't find the container with id 8412394f26d59469866df6ded7a8bb9582ec3aeaa16a12d81b58d7f18c099e36 Jul 21 04:15:30.077143 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:30.077112 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-576mf" event={"ID":"0c9ba075-91dd-440e-ba17-c235f89b3e9f","Type":"ContainerStarted","Data":"8412394f26d59469866df6ded7a8bb9582ec3aeaa16a12d81b58d7f18c099e36"} Jul 21 04:15:31.083565 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:31.083469 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" event={"ID":"5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1","Type":"ContainerStarted","Data":"e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33"} Jul 21 04:15:31.083565 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:31.083493 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" podUID="5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1" containerName="registry-server" containerID="cri-o://e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33" gracePeriod=2 Jul 21 04:15:31.085792 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:31.085764 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-576mf" event={"ID":"0c9ba075-91dd-440e-ba17-c235f89b3e9f","Type":"ContainerStarted","Data":"142b6cd86d90581df76713e09fdb661a66b3b813c7858d030fe5385654167010"} Jul 21 04:15:31.101046 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:31.101003 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" podStartSLOduration=0.929729528 podStartE2EDuration="3.100989395s" totalImagesPullingTime="2.171259867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:15:31.099059295 +0000 UTC m=+376.498392055" watchObservedRunningTime="2026-07-21 04:15:31.100989395 +0000 UTC m=+376.500322153" Jul 21 04:15:31.115423 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:31.115381 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-576mf" podStartSLOduration=1.518206685 podStartE2EDuration="3.115365841s" totalImagesPullingTime="1.597159156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:15:31.113739401 +0000 UTC m=+376.513072155" watchObservedRunningTime="2026-07-21 04:15:31.115365841 +0000 UTC m=+376.514698596" Jul 21 04:15:31.330497 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:31.330469 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" Jul 21 04:15:31.408429 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:31.408401 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrgpr\" (UniqueName: \"kubernetes.io/projected/5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1-kube-api-access-wrgpr\") pod \"5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1\" (UID: \"5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1\") " Jul 21 04:15:31.410621 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:31.410599 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1-kube-api-access-wrgpr" (OuterVolumeSpecName: "kube-api-access-wrgpr") pod "5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1" (UID: "5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1"). InnerVolumeSpecName "kube-api-access-wrgpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:15:31.509120 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:31.509040 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrgpr\" (UniqueName: \"kubernetes.io/projected/5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1-kube-api-access-wrgpr\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:32.091997 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:32.091959 2567 generic.go:356] "Generic (PLEG): container finished" podID="5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1" containerID="e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33" exitCode=0 Jul 21 04:15:32.092439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:32.092006 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" Jul 21 04:15:32.092439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:32.092039 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" event={"ID":"5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1","Type":"ContainerDied","Data":"e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33"} Jul 21 04:15:32.092439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:32.092100 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7dp58" event={"ID":"5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1","Type":"ContainerDied","Data":"2d5dbec166f08f9382bcc7cd907289b139de4fdb7d5baff4568e3ff955789208"} Jul 21 04:15:32.092439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:32.092122 2567 scope.go:117] "RemoveContainer" containerID="e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33" Jul 21 04:15:32.101126 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:32.101107 2567 scope.go:117] "RemoveContainer" containerID="e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33" Jul 21 04:15:32.101417 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:15:32.101398 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33\": container with ID starting with e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33 not found: ID does not exist" containerID="e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33" Jul 21 04:15:32.101494 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:32.101426 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33"} err="failed to get container status \"e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33\": rpc error: code = NotFound desc = could not find container \"e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33\": container with ID starting with e0945808745b0f71190f09f88bb1959f5d7bd1da5560b2f1e98c8553d8cd7f33 not found: ID does not exist" Jul 21 04:15:32.115353 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:32.115319 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7dp58"] Jul 21 04:15:32.119376 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:32.119331 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7dp58"] Jul 21 04:15:33.265454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:33.265421 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1" path="/var/lib/kubelet/pods/5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1/volumes" Jul 21 04:15:39.140374 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:39.140319 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-576mf" Jul 21 04:15:39.140906 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:39.140390 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-576mf" Jul 21 04:15:39.162937 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:39.162905 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-576mf" Jul 21 04:15:40.159524 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:40.159492 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-576mf" Jul 21 04:15:43.629028 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.628990 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb"] Jul 21 04:15:43.629462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.629319 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1" containerName="registry-server" Jul 21 04:15:43.629462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.629330 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1" containerName="registry-server" Jul 21 04:15:43.629462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.629417 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f1284a2-f5ae-4384-a1a2-0aa3d0d221b1" containerName="registry-server" Jul 21 04:15:43.632167 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.632148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.635007 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.634981 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wx5hb\"" Jul 21 04:15:43.641630 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.641603 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb"] Jul 21 04:15:43.714756 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.714728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.714902 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.714774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhthc\" (UniqueName: \"kubernetes.io/projected/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-kube-api-access-mhthc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.714902 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.714843 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.815212 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.815182 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.815322 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.815227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhthc\" (UniqueName: \"kubernetes.io/projected/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-kube-api-access-mhthc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.815322 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.815261 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.815573 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.815553 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.815627 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.815607 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.824784 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.824766 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhthc\" (UniqueName: \"kubernetes.io/projected/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-kube-api-access-mhthc\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:43.944200 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:43.944140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:44.096923 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.096899 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb"] Jul 21 04:15:44.099279 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:15:44.099252 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc785cd48_ef6a_476b_9e4d_ea8a4bf994cb.slice/crio-af93887ea1e2caa563b1ae00a8bf80d8152992bc021347db1eb4d909397f66b8 WatchSource:0}: Error finding container af93887ea1e2caa563b1ae00a8bf80d8152992bc021347db1eb4d909397f66b8: Status 404 returned error can't find the container with id af93887ea1e2caa563b1ae00a8bf80d8152992bc021347db1eb4d909397f66b8 Jul 21 04:15:44.157586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.157558 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" event={"ID":"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb","Type":"ContainerStarted","Data":"af93887ea1e2caa563b1ae00a8bf80d8152992bc021347db1eb4d909397f66b8"} Jul 21 04:15:44.419576 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.419544 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq"] Jul 21 04:15:44.422838 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.422820 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.431199 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.431172 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq"] Jul 21 04:15:44.522110 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.522080 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.522265 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.522114 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.522265 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.522188 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hwj\" (UniqueName: \"kubernetes.io/projected/e67cda2d-9360-469e-b915-0d3b3568e60f-kube-api-access-b9hwj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.622547 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.622518 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.622547 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.622548 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.622708 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.622583 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hwj\" (UniqueName: \"kubernetes.io/projected/e67cda2d-9360-469e-b915-0d3b3568e60f-kube-api-access-b9hwj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.622906 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.622888 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.622956 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.622940 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.631043 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.631020 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hwj\" (UniqueName: \"kubernetes.io/projected/e67cda2d-9360-469e-b915-0d3b3568e60f-kube-api-access-b9hwj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.731983 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.731956 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:44.860186 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:44.860163 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq"] Jul 21 04:15:44.861908 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:15:44.861880 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67cda2d_9360_469e_b915_0d3b3568e60f.slice/crio-436c53ef325f5ef967c81c5ec0d9cb97ad6c6179886c437b697e46d656b73f6a WatchSource:0}: Error finding container 436c53ef325f5ef967c81c5ec0d9cb97ad6c6179886c437b697e46d656b73f6a: Status 404 returned error can't find the container with id 436c53ef325f5ef967c81c5ec0d9cb97ad6c6179886c437b697e46d656b73f6a Jul 21 04:15:45.019966 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.019935 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4"] Jul 21 04:15:45.023241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.023226 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.025588 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.025566 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4629\" (UniqueName: \"kubernetes.io/projected/a1dac966-414e-4aee-9d1c-efe3c5c80e81-kube-api-access-s4629\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.025669 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.025628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.025739 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.025682 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.031800 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.031777 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4"] Jul 21 04:15:45.126389 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.126364 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4629\" (UniqueName: \"kubernetes.io/projected/a1dac966-414e-4aee-9d1c-efe3c5c80e81-kube-api-access-s4629\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.126516 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.126416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.126516 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.126463 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.126846 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.126825 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.126881 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.126840 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.134984 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.134954 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4629\" (UniqueName: \"kubernetes.io/projected/a1dac966-414e-4aee-9d1c-efe3c5c80e81-kube-api-access-s4629\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.163087 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.163059 2567 generic.go:356] "Generic (PLEG): container finished" podID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerID="931b8cf8eb94e4f71bd34e66e3ba07b9a58519f8104290a80113d3f2c76a25bf" exitCode=0 Jul 21 04:15:45.163200 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.163151 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" event={"ID":"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb","Type":"ContainerDied","Data":"931b8cf8eb94e4f71bd34e66e3ba07b9a58519f8104290a80113d3f2c76a25bf"} Jul 21 04:15:45.164834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.164812 2567 generic.go:356] "Generic (PLEG): container finished" podID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerID="6b478d4b6e9c484859c066bfbb8932c4a68d8df7fe231738493cd43701379ec9" exitCode=0 Jul 21 04:15:45.164928 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.164852 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" event={"ID":"e67cda2d-9360-469e-b915-0d3b3568e60f","Type":"ContainerDied","Data":"6b478d4b6e9c484859c066bfbb8932c4a68d8df7fe231738493cd43701379ec9"} Jul 21 04:15:45.164928 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.164884 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" event={"ID":"e67cda2d-9360-469e-b915-0d3b3568e60f","Type":"ContainerStarted","Data":"436c53ef325f5ef967c81c5ec0d9cb97ad6c6179886c437b697e46d656b73f6a"} Jul 21 04:15:45.333598 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.333528 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:45.459854 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.459827 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4"] Jul 21 04:15:45.461309 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:15:45.461283 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1dac966_414e_4aee_9d1c_efe3c5c80e81.slice/crio-fbfd9f2c9bd2c4037f34c578a3dc78b08c340e37fa2a2968c2f4c237ae4151fb WatchSource:0}: Error finding container fbfd9f2c9bd2c4037f34c578a3dc78b08c340e37fa2a2968c2f4c237ae4151fb: Status 404 returned error can't find the container with id fbfd9f2c9bd2c4037f34c578a3dc78b08c340e37fa2a2968c2f4c237ae4151fb Jul 21 04:15:45.618139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.618080 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl"] Jul 21 04:15:45.621554 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.621536 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.630040 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.629897 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5fbk\" (UniqueName: \"kubernetes.io/projected/3b0dffcc-329b-45ff-be93-41b3b52eff4a-kube-api-access-k5fbk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.630040 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.629986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.630397 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.630060 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.631715 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.631670 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl"] Jul 21 04:15:45.730742 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.730710 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.730912 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.730769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5fbk\" (UniqueName: \"kubernetes.io/projected/3b0dffcc-329b-45ff-be93-41b3b52eff4a-kube-api-access-k5fbk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.730912 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.730808 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.731180 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.731156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.731217 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.731167 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.742140 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.742114 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5fbk\" (UniqueName: \"kubernetes.io/projected/3b0dffcc-329b-45ff-be93-41b3b52eff4a-kube-api-access-k5fbk\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:45.933477 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:45.933416 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:46.065349 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.065315 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl"] Jul 21 04:15:46.067187 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:15:46.067163 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0dffcc_329b_45ff_be93_41b3b52eff4a.slice/crio-60dd850d2e163eab22db72be9c9fe2f186e0e2fcb7b2bb3336c2dd2dfb9e1f45 WatchSource:0}: Error finding container 60dd850d2e163eab22db72be9c9fe2f186e0e2fcb7b2bb3336c2dd2dfb9e1f45: Status 404 returned error can't find the container with id 60dd850d2e163eab22db72be9c9fe2f186e0e2fcb7b2bb3336c2dd2dfb9e1f45 Jul 21 04:15:46.171807 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.171767 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" event={"ID":"3b0dffcc-329b-45ff-be93-41b3b52eff4a","Type":"ContainerStarted","Data":"32e512bdcb71da81c3b9e752e012c47cedc44eabd36055c67f20131c1e60420f"} Jul 21 04:15:46.171944 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.171850 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" event={"ID":"3b0dffcc-329b-45ff-be93-41b3b52eff4a","Type":"ContainerStarted","Data":"60dd850d2e163eab22db72be9c9fe2f186e0e2fcb7b2bb3336c2dd2dfb9e1f45"} Jul 21 04:15:46.173689 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.173658 2567 generic.go:356] "Generic (PLEG): container finished" podID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerID="00583c9b83165839df6cd7c1102692052bdd246be718b5a77dc33b0199ec221f" exitCode=0 Jul 21 04:15:46.173791 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.173775 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" event={"ID":"e67cda2d-9360-469e-b915-0d3b3568e60f","Type":"ContainerDied","Data":"00583c9b83165839df6cd7c1102692052bdd246be718b5a77dc33b0199ec221f"} Jul 21 04:15:46.175428 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.175400 2567 generic.go:356] "Generic (PLEG): container finished" podID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerID="51adb7e3b978fa8ffb4177bcc59c1e7b96a73b71f27434679f2cf59dd6684da2" exitCode=0 Jul 21 04:15:46.175502 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.175453 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" event={"ID":"a1dac966-414e-4aee-9d1c-efe3c5c80e81","Type":"ContainerDied","Data":"51adb7e3b978fa8ffb4177bcc59c1e7b96a73b71f27434679f2cf59dd6684da2"} Jul 21 04:15:46.175562 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.175515 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" event={"ID":"a1dac966-414e-4aee-9d1c-efe3c5c80e81","Type":"ContainerStarted","Data":"fbfd9f2c9bd2c4037f34c578a3dc78b08c340e37fa2a2968c2f4c237ae4151fb"} Jul 21 04:15:46.177582 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.177559 2567 generic.go:356] "Generic (PLEG): container finished" podID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerID="78b7fdff11eae8b9e7d780f794979b09382887e0234d39f805087b0e213bc70e" exitCode=0 Jul 21 04:15:46.177669 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:46.177625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" event={"ID":"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb","Type":"ContainerDied","Data":"78b7fdff11eae8b9e7d780f794979b09382887e0234d39f805087b0e213bc70e"} Jul 21 04:15:47.183939 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:47.183908 2567 generic.go:356] "Generic (PLEG): container finished" podID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerID="ea3d4f526a816ab94256f04963a1061a5ffa45b4c9f65b1d50421d4ef092752b" exitCode=0 Jul 21 04:15:47.184386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:47.183996 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" event={"ID":"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb","Type":"ContainerDied","Data":"ea3d4f526a816ab94256f04963a1061a5ffa45b4c9f65b1d50421d4ef092752b"} Jul 21 04:15:47.185600 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:47.185565 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerID="32e512bdcb71da81c3b9e752e012c47cedc44eabd36055c67f20131c1e60420f" exitCode=0 Jul 21 04:15:47.185695 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:47.185650 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" event={"ID":"3b0dffcc-329b-45ff-be93-41b3b52eff4a","Type":"ContainerDied","Data":"32e512bdcb71da81c3b9e752e012c47cedc44eabd36055c67f20131c1e60420f"} Jul 21 04:15:47.187757 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:47.187738 2567 generic.go:356] "Generic (PLEG): container finished" podID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerID="9a5f9a9347a71437c078c94a72948bc54d32cb0e2c593ca6c4543dbd79cf6438" exitCode=0 Jul 21 04:15:47.187838 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:47.187812 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" event={"ID":"e67cda2d-9360-469e-b915-0d3b3568e60f","Type":"ContainerDied","Data":"9a5f9a9347a71437c078c94a72948bc54d32cb0e2c593ca6c4543dbd79cf6438"} Jul 21 04:15:47.189698 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:47.189681 2567 generic.go:356] "Generic (PLEG): container finished" podID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerID="c5442c7828aa9345f6d55406c76dd683110b2b845cc4a759c1e045ca8195f8f9" exitCode=0 Jul 21 04:15:47.189803 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:47.189730 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" event={"ID":"a1dac966-414e-4aee-9d1c-efe3c5c80e81","Type":"ContainerDied","Data":"c5442c7828aa9345f6d55406c76dd683110b2b845cc4a759c1e045ca8195f8f9"} Jul 21 04:15:48.199751 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.199715 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerID="adfcc7919f6e76a15344f680eb98ad18bc46ef9ae3bb5df8aa8d5a927fc00f0e" exitCode=0 Jul 21 04:15:48.200160 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.199773 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" event={"ID":"3b0dffcc-329b-45ff-be93-41b3b52eff4a","Type":"ContainerDied","Data":"adfcc7919f6e76a15344f680eb98ad18bc46ef9ae3bb5df8aa8d5a927fc00f0e"} Jul 21 04:15:48.202021 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.201998 2567 generic.go:356] "Generic (PLEG): container finished" podID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerID="101b18a4f75ac75e23b4a9ddc7e1eb2846dc3fdd2661b968156c756551c7fae9" exitCode=0 Jul 21 04:15:48.202125 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.202104 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" event={"ID":"a1dac966-414e-4aee-9d1c-efe3c5c80e81","Type":"ContainerDied","Data":"101b18a4f75ac75e23b4a9ddc7e1eb2846dc3fdd2661b968156c756551c7fae9"} Jul 21 04:15:48.350445 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.350417 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:48.411476 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.411443 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:48.449195 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.449163 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-bundle\") pod \"e67cda2d-9360-469e-b915-0d3b3568e60f\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " Jul 21 04:15:48.449331 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.449233 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhthc\" (UniqueName: \"kubernetes.io/projected/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-kube-api-access-mhthc\") pod \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " Jul 21 04:15:48.449331 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.449270 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-bundle\") pod \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " Jul 21 04:15:48.449331 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.449320 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9hwj\" (UniqueName: \"kubernetes.io/projected/e67cda2d-9360-469e-b915-0d3b3568e60f-kube-api-access-b9hwj\") pod \"e67cda2d-9360-469e-b915-0d3b3568e60f\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " Jul 21 04:15:48.449504 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.449392 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-util\") pod \"e67cda2d-9360-469e-b915-0d3b3568e60f\" (UID: \"e67cda2d-9360-469e-b915-0d3b3568e60f\") " Jul 21 04:15:48.449504 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.449420 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-util\") pod \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\" (UID: \"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb\") " Jul 21 04:15:48.449831 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.449802 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-bundle" (OuterVolumeSpecName: "bundle") pod "e67cda2d-9360-469e-b915-0d3b3568e60f" (UID: "e67cda2d-9360-469e-b915-0d3b3568e60f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:15:48.450090 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.450069 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-bundle" (OuterVolumeSpecName: "bundle") pod "c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" (UID: "c785cd48-ef6a-476b-9e4d-ea8a4bf994cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:15:48.451420 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.451389 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-kube-api-access-mhthc" (OuterVolumeSpecName: "kube-api-access-mhthc") pod "c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" (UID: "c785cd48-ef6a-476b-9e4d-ea8a4bf994cb"). InnerVolumeSpecName "kube-api-access-mhthc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:15:48.451691 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.451659 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67cda2d-9360-469e-b915-0d3b3568e60f-kube-api-access-b9hwj" (OuterVolumeSpecName: "kube-api-access-b9hwj") pod "e67cda2d-9360-469e-b915-0d3b3568e60f" (UID: "e67cda2d-9360-469e-b915-0d3b3568e60f"). InnerVolumeSpecName "kube-api-access-b9hwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:15:48.454668 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.454642 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-util" (OuterVolumeSpecName: "util") pod "e67cda2d-9360-469e-b915-0d3b3568e60f" (UID: "e67cda2d-9360-469e-b915-0d3b3568e60f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:15:48.457174 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.457155 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-util" (OuterVolumeSpecName: "util") pod "c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" (UID: "c785cd48-ef6a-476b-9e4d-ea8a4bf994cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:15:48.550052 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.550025 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-util\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:48.550052 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.550050 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-util\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:48.550241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.550062 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e67cda2d-9360-469e-b915-0d3b3568e60f-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:48.550241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.550075 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhthc\" (UniqueName: \"kubernetes.io/projected/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-kube-api-access-mhthc\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:48.550241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.550088 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c785cd48-ef6a-476b-9e4d-ea8a4bf994cb-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:48.550241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:48.550103 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9hwj\" (UniqueName: \"kubernetes.io/projected/e67cda2d-9360-469e-b915-0d3b3568e60f-kube-api-access-b9hwj\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:49.209740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.209711 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" Jul 21 04:15:49.209740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.209733 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb" event={"ID":"c785cd48-ef6a-476b-9e4d-ea8a4bf994cb","Type":"ContainerDied","Data":"af93887ea1e2caa563b1ae00a8bf80d8152992bc021347db1eb4d909397f66b8"} Jul 21 04:15:49.210215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.209825 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af93887ea1e2caa563b1ae00a8bf80d8152992bc021347db1eb4d909397f66b8" Jul 21 04:15:49.212001 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.211973 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerID="bd80e259272e4361d58e10b3b810dfb0e2189844b5184ce847b1f5fbdbb94c1d" exitCode=0 Jul 21 04:15:49.212136 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.212050 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" event={"ID":"3b0dffcc-329b-45ff-be93-41b3b52eff4a","Type":"ContainerDied","Data":"bd80e259272e4361d58e10b3b810dfb0e2189844b5184ce847b1f5fbdbb94c1d"} Jul 21 04:15:49.213861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.213837 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" event={"ID":"e67cda2d-9360-469e-b915-0d3b3568e60f","Type":"ContainerDied","Data":"436c53ef325f5ef967c81c5ec0d9cb97ad6c6179886c437b697e46d656b73f6a"} Jul 21 04:15:49.213969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.213860 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq" Jul 21 04:15:49.213969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.213873 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="436c53ef325f5ef967c81c5ec0d9cb97ad6c6179886c437b697e46d656b73f6a" Jul 21 04:15:49.347506 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.347482 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:49.458583 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.458555 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4629\" (UniqueName: \"kubernetes.io/projected/a1dac966-414e-4aee-9d1c-efe3c5c80e81-kube-api-access-s4629\") pod \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " Jul 21 04:15:49.458726 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.458626 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-util\") pod \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " Jul 21 04:15:49.458726 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.458651 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-bundle\") pod \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\" (UID: \"a1dac966-414e-4aee-9d1c-efe3c5c80e81\") " Jul 21 04:15:49.459189 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.459161 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-bundle" (OuterVolumeSpecName: "bundle") pod "a1dac966-414e-4aee-9d1c-efe3c5c80e81" (UID: "a1dac966-414e-4aee-9d1c-efe3c5c80e81"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:15:49.460693 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.460668 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1dac966-414e-4aee-9d1c-efe3c5c80e81-kube-api-access-s4629" (OuterVolumeSpecName: "kube-api-access-s4629") pod "a1dac966-414e-4aee-9d1c-efe3c5c80e81" (UID: "a1dac966-414e-4aee-9d1c-efe3c5c80e81"). InnerVolumeSpecName "kube-api-access-s4629". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:15:49.463714 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.463695 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-util" (OuterVolumeSpecName: "util") pod "a1dac966-414e-4aee-9d1c-efe3c5c80e81" (UID: "a1dac966-414e-4aee-9d1c-efe3c5c80e81"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:15:49.559636 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.559564 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-util\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:49.559636 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.559598 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1dac966-414e-4aee-9d1c-efe3c5c80e81-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:49.559636 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:49.559608 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4629\" (UniqueName: \"kubernetes.io/projected/a1dac966-414e-4aee-9d1c-efe3c5c80e81-kube-api-access-s4629\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:50.220863 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.220838 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" Jul 21 04:15:50.221210 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.220879 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4" event={"ID":"a1dac966-414e-4aee-9d1c-efe3c5c80e81","Type":"ContainerDied","Data":"fbfd9f2c9bd2c4037f34c578a3dc78b08c340e37fa2a2968c2f4c237ae4151fb"} Jul 21 04:15:50.221210 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.220925 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbfd9f2c9bd2c4037f34c578a3dc78b08c340e37fa2a2968c2f4c237ae4151fb" Jul 21 04:15:50.357060 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.357037 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:15:50.365452 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.365428 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5fbk\" (UniqueName: \"kubernetes.io/projected/3b0dffcc-329b-45ff-be93-41b3b52eff4a-kube-api-access-k5fbk\") pod \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " Jul 21 04:15:50.367290 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.367271 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0dffcc-329b-45ff-be93-41b3b52eff4a-kube-api-access-k5fbk" (OuterVolumeSpecName: "kube-api-access-k5fbk") pod "3b0dffcc-329b-45ff-be93-41b3b52eff4a" (UID: "3b0dffcc-329b-45ff-be93-41b3b52eff4a"). InnerVolumeSpecName "kube-api-access-k5fbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:15:50.466571 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.466546 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-util\") pod \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " Jul 21 04:15:50.466713 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.466593 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-bundle\") pod \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\" (UID: \"3b0dffcc-329b-45ff-be93-41b3b52eff4a\") " Jul 21 04:15:50.466840 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.466824 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5fbk\" (UniqueName: \"kubernetes.io/projected/3b0dffcc-329b-45ff-be93-41b3b52eff4a-kube-api-access-k5fbk\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:50.467123 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.467102 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-bundle" (OuterVolumeSpecName: "bundle") pod "3b0dffcc-329b-45ff-be93-41b3b52eff4a" (UID: "3b0dffcc-329b-45ff-be93-41b3b52eff4a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:15:50.471864 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.471822 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-util" (OuterVolumeSpecName: "util") pod "3b0dffcc-329b-45ff-be93-41b3b52eff4a" (UID: "3b0dffcc-329b-45ff-be93-41b3b52eff4a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:15:50.567258 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.567225 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-util\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:50.567258 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:50.567253 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b0dffcc-329b-45ff-be93-41b3b52eff4a-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:15:51.227333 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:51.227300 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" event={"ID":"3b0dffcc-329b-45ff-be93-41b3b52eff4a","Type":"ContainerDied","Data":"60dd850d2e163eab22db72be9c9fe2f186e0e2fcb7b2bb3336c2dd2dfb9e1f45"} Jul 21 04:15:51.227725 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:51.227378 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60dd850d2e163eab22db72be9c9fe2f186e0e2fcb7b2bb3336c2dd2dfb9e1f45" Jul 21 04:15:51.227725 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:15:51.227312 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl" Jul 21 04:16:00.674663 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.674624 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd"] Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.674937 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerName="pull" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.674947 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerName="pull" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.674960 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerName="util" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.674966 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerName="util" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.674974 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerName="pull" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.674979 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerName="pull" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.674987 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerName="util" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.674992 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerName="util" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675001 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerName="pull" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675006 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerName="pull" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675011 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerName="extract" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675016 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerName="extract" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675027 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerName="util" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675032 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerName="util" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675039 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerName="extract" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675043 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerName="extract" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675053 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerName="util" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675058 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerName="util" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675064 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerName="extract" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675069 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerName="extract" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675075 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerName="extract" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675081 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerName="extract" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675088 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerName="pull" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675093 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerName="pull" Jul 21 04:16:00.675128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675142 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67cda2d-9360-469e-b915-0d3b3568e60f" containerName="extract" Jul 21 04:16:00.675977 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675150 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1dac966-414e-4aee-9d1c-efe3c5c80e81" containerName="extract" Jul 21 04:16:00.675977 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675158 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c785cd48-ef6a-476b-9e4d-ea8a4bf994cb" containerName="extract" Jul 21 04:16:00.675977 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.675165 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0dffcc-329b-45ff-be93-41b3b52eff4a" containerName="extract" Jul 21 04:16:00.678474 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.678452 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" Jul 21 04:16:00.681035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.681011 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zs96g\"" Jul 21 04:16:00.681152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.681035 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 21 04:16:00.689584 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.689562 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd"] Jul 21 04:16:00.746739 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.746713 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknmh\" (UniqueName: \"kubernetes.io/projected/3ecc4112-39f8-4744-9967-8c73df953f92-kube-api-access-pknmh\") pod \"dns-operator-controller-manager-864dd78c4c-rtsvd\" (UID: \"3ecc4112-39f8-4744-9967-8c73df953f92\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" Jul 21 04:16:00.847939 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.847900 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknmh\" (UniqueName: \"kubernetes.io/projected/3ecc4112-39f8-4744-9967-8c73df953f92-kube-api-access-pknmh\") pod \"dns-operator-controller-manager-864dd78c4c-rtsvd\" (UID: \"3ecc4112-39f8-4744-9967-8c73df953f92\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" Jul 21 04:16:00.856867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.856838 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknmh\" (UniqueName: \"kubernetes.io/projected/3ecc4112-39f8-4744-9967-8c73df953f92-kube-api-access-pknmh\") pod \"dns-operator-controller-manager-864dd78c4c-rtsvd\" (UID: \"3ecc4112-39f8-4744-9967-8c73df953f92\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" Jul 21 04:16:00.989236 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:00.989164 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" Jul 21 04:16:01.126909 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:01.126874 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd"] Jul 21 04:16:01.129944 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:16:01.129919 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecc4112_39f8_4744_9967_8c73df953f92.slice/crio-8aca753557cbde9ed47acce4e63c61fdaed3b2903325368bf210e6137ee6f281 WatchSource:0}: Error finding container 8aca753557cbde9ed47acce4e63c61fdaed3b2903325368bf210e6137ee6f281: Status 404 returned error can't find the container with id 8aca753557cbde9ed47acce4e63c61fdaed3b2903325368bf210e6137ee6f281 Jul 21 04:16:01.274746 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:01.274709 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" event={"ID":"3ecc4112-39f8-4744-9967-8c73df953f92","Type":"ContainerStarted","Data":"8aca753557cbde9ed47acce4e63c61fdaed3b2903325368bf210e6137ee6f281"} Jul 21 04:16:03.286699 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:03.286663 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" event={"ID":"3ecc4112-39f8-4744-9967-8c73df953f92","Type":"ContainerStarted","Data":"c3581427e482b2808b70e75b7d2debd42e02aacb4ad4445d38a392b2c172e2f8"} Jul 21 04:16:03.287030 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:03.286772 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" Jul 21 04:16:03.304357 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:03.304292 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" podStartSLOduration=1.222574927 podStartE2EDuration="3.304280135s" totalImagesPullingTime="2.081705208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:16:03.3028483 +0000 UTC m=+408.702181059" watchObservedRunningTime="2026-07-21 04:16:03.304280135 +0000 UTC m=+408.703612893" Jul 21 04:16:05.707420 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:05.707390 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts"] Jul 21 04:16:05.710919 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:05.710902 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:05.714154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:05.714135 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kkqmh\"" Jul 21 04:16:05.724711 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:05.724690 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts"] Jul 21 04:16:05.791048 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:05.791015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwljz\" (UniqueName: \"kubernetes.io/projected/2e92db26-372e-42cf-b341-98492407bd24-kube-api-access-jwljz\") pod \"limitador-operator-controller-manager-859d7b78bd-dvkts\" (UID: \"2e92db26-372e-42cf-b341-98492407bd24\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:05.892076 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:05.892039 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwljz\" (UniqueName: \"kubernetes.io/projected/2e92db26-372e-42cf-b341-98492407bd24-kube-api-access-jwljz\") pod \"limitador-operator-controller-manager-859d7b78bd-dvkts\" (UID: \"2e92db26-372e-42cf-b341-98492407bd24\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:05.901279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:05.901249 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwljz\" (UniqueName: \"kubernetes.io/projected/2e92db26-372e-42cf-b341-98492407bd24-kube-api-access-jwljz\") pod \"limitador-operator-controller-manager-859d7b78bd-dvkts\" (UID: \"2e92db26-372e-42cf-b341-98492407bd24\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:06.020632 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:06.020602 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:06.157668 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:06.157630 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts"] Jul 21 04:16:06.158985 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:16:06.158953 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e92db26_372e_42cf_b341_98492407bd24.slice/crio-0218c29250cb23fcd8cc8e48c99df6e32391c0613bc99a6d84ffe4f485f5729f WatchSource:0}: Error finding container 0218c29250cb23fcd8cc8e48c99df6e32391c0613bc99a6d84ffe4f485f5729f: Status 404 returned error can't find the container with id 0218c29250cb23fcd8cc8e48c99df6e32391c0613bc99a6d84ffe4f485f5729f Jul 21 04:16:06.305472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:06.305391 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" event={"ID":"2e92db26-372e-42cf-b341-98492407bd24","Type":"ContainerStarted","Data":"0218c29250cb23fcd8cc8e48c99df6e32391c0613bc99a6d84ffe4f485f5729f"} Jul 21 04:16:08.318050 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:08.318003 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" event={"ID":"2e92db26-372e-42cf-b341-98492407bd24","Type":"ContainerStarted","Data":"20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1"} Jul 21 04:16:08.318507 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:08.318102 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:08.334751 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:08.334708 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" podStartSLOduration=2.025402554 podStartE2EDuration="3.334694844s" totalImagesPullingTime="1.30929229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:16:08.33389349 +0000 UTC m=+413.733226248" watchObservedRunningTime="2026-07-21 04:16:08.334694844 +0000 UTC m=+413.734027601" Jul 21 04:16:10.574508 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:10.574476 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt"] Jul 21 04:16:10.577864 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:10.577847 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" Jul 21 04:16:10.581033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:10.581008 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8zszc\"" Jul 21 04:16:10.588824 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:10.588803 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt"] Jul 21 04:16:10.631121 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:10.631099 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22459\" (UniqueName: \"kubernetes.io/projected/61a06832-b305-4be3-95b9-78e351681fa0-kube-api-access-22459\") pod \"authorino-operator-75cc8d5cb7-2zzxt\" (UID: \"61a06832-b305-4be3-95b9-78e351681fa0\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" Jul 21 04:16:10.732399 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:10.732373 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22459\" (UniqueName: \"kubernetes.io/projected/61a06832-b305-4be3-95b9-78e351681fa0-kube-api-access-22459\") pod \"authorino-operator-75cc8d5cb7-2zzxt\" (UID: \"61a06832-b305-4be3-95b9-78e351681fa0\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" Jul 21 04:16:10.742228 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:10.742202 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22459\" (UniqueName: \"kubernetes.io/projected/61a06832-b305-4be3-95b9-78e351681fa0-kube-api-access-22459\") pod \"authorino-operator-75cc8d5cb7-2zzxt\" (UID: \"61a06832-b305-4be3-95b9-78e351681fa0\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" Jul 21 04:16:10.888743 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:10.888678 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" Jul 21 04:16:11.017215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:11.017189 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt"] Jul 21 04:16:11.019367 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:16:11.019316 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a06832_b305_4be3_95b9_78e351681fa0.slice/crio-c023353bb10b0218b7ac65406ed205d7328462faa990b0beb7fd119529c22dab WatchSource:0}: Error finding container c023353bb10b0218b7ac65406ed205d7328462faa990b0beb7fd119529c22dab: Status 404 returned error can't find the container with id c023353bb10b0218b7ac65406ed205d7328462faa990b0beb7fd119529c22dab Jul 21 04:16:11.334194 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:11.334152 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" event={"ID":"61a06832-b305-4be3-95b9-78e351681fa0","Type":"ContainerStarted","Data":"c023353bb10b0218b7ac65406ed205d7328462faa990b0beb7fd119529c22dab"} Jul 21 04:16:13.346676 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:13.346645 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" event={"ID":"61a06832-b305-4be3-95b9-78e351681fa0","Type":"ContainerStarted","Data":"51a89f464e861a57cc25a48ac224b6b94b3997a7f1cff67b9b2d5559668b4917"} Jul 21 04:16:13.347029 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:13.346773 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" Jul 21 04:16:13.363605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:13.363559 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" podStartSLOduration=1.192447232 podStartE2EDuration="3.36354601s" totalImagesPullingTime="2.171098778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:16:13.362500778 +0000 UTC m=+418.761833534" watchObservedRunningTime="2026-07-21 04:16:13.36354601 +0000 UTC m=+418.762878769" Jul 21 04:16:14.294939 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:14.294904 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-rtsvd" Jul 21 04:16:16.127979 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.127944 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg"] Jul 21 04:16:16.132761 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.132741 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:16.135278 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.135262 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jnblz\"" Jul 21 04:16:16.144628 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.144605 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg"] Jul 21 04:16:16.276882 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.276830 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53b8d99-c7a1-4830-86e4-e87a91283cdb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" (UID: \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:16.277030 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.276975 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h6f6\" (UniqueName: \"kubernetes.io/projected/a53b8d99-c7a1-4830-86e4-e87a91283cdb-kube-api-access-8h6f6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" (UID: \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:16.378047 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.377973 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h6f6\" (UniqueName: \"kubernetes.io/projected/a53b8d99-c7a1-4830-86e4-e87a91283cdb-kube-api-access-8h6f6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" (UID: \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:16.378047 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.378017 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53b8d99-c7a1-4830-86e4-e87a91283cdb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" (UID: \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:16.378411 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.378382 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53b8d99-c7a1-4830-86e4-e87a91283cdb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" (UID: \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:16.387305 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.387281 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h6f6\" (UniqueName: \"kubernetes.io/projected/a53b8d99-c7a1-4830-86e4-e87a91283cdb-kube-api-access-8h6f6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" (UID: \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:16.443535 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.443512 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:16.580645 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:16.580613 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg"] Jul 21 04:16:16.584149 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:16:16.584124 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53b8d99_c7a1_4830_86e4_e87a91283cdb.slice/crio-e8a8cf5602eb11e01be4bbab44fedcaa585f0fc2acea4b1b7a621f663160fd50 WatchSource:0}: Error finding container e8a8cf5602eb11e01be4bbab44fedcaa585f0fc2acea4b1b7a621f663160fd50: Status 404 returned error can't find the container with id e8a8cf5602eb11e01be4bbab44fedcaa585f0fc2acea4b1b7a621f663160fd50 Jul 21 04:16:17.368438 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:17.368404 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" event={"ID":"a53b8d99-c7a1-4830-86e4-e87a91283cdb","Type":"ContainerStarted","Data":"e8a8cf5602eb11e01be4bbab44fedcaa585f0fc2acea4b1b7a621f663160fd50"} Jul 21 04:16:18.144599 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.144561 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-964f5b447-8hjqx"] Jul 21 04:16:18.150074 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.150053 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.157438 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.157412 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-964f5b447-8hjqx"] Jul 21 04:16:18.294179 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.294145 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-service-ca\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.294368 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.294202 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-trusted-ca-bundle\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.294368 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.294258 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6412c0e-d008-46d9-9296-cfdc4ebe666e-console-serving-cert\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.294368 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.294282 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qzk\" (UniqueName: \"kubernetes.io/projected/e6412c0e-d008-46d9-9296-cfdc4ebe666e-kube-api-access-p5qzk\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.294368 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.294312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6412c0e-d008-46d9-9296-cfdc4ebe666e-console-oauth-config\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.294560 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.294406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-console-config\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.294560 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.294467 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-oauth-serving-cert\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.395819 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.395749 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-oauth-serving-cert\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.395838 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-service-ca\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.395881 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-trusted-ca-bundle\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.395908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6412c0e-d008-46d9-9296-cfdc4ebe666e-console-serving-cert\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.395931 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qzk\" (UniqueName: \"kubernetes.io/projected/e6412c0e-d008-46d9-9296-cfdc4ebe666e-kube-api-access-p5qzk\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.395956 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6412c0e-d008-46d9-9296-cfdc4ebe666e-console-oauth-config\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.395981 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-console-config\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396665 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.396637 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-service-ca\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396805 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.396699 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-oauth-serving-cert\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396805 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.396774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-console-config\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.396979 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.396960 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6412c0e-d008-46d9-9296-cfdc4ebe666e-trusted-ca-bundle\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.398521 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.398491 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6412c0e-d008-46d9-9296-cfdc4ebe666e-console-oauth-config\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.398774 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.398752 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6412c0e-d008-46d9-9296-cfdc4ebe666e-console-serving-cert\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.404410 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.404390 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qzk\" (UniqueName: \"kubernetes.io/projected/e6412c0e-d008-46d9-9296-cfdc4ebe666e-kube-api-access-p5qzk\") pod \"console-964f5b447-8hjqx\" (UID: \"e6412c0e-d008-46d9-9296-cfdc4ebe666e\") " pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.463316 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.463288 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:18.626249 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:18.626216 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-964f5b447-8hjqx"] Jul 21 04:16:18.627332 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:16:18.627306 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6412c0e_d008_46d9_9296_cfdc4ebe666e.slice/crio-29b436635de40aefc0f240ff79ac016828687067bb9ab82a73e3b2b541bad5ed WatchSource:0}: Error finding container 29b436635de40aefc0f240ff79ac016828687067bb9ab82a73e3b2b541bad5ed: Status 404 returned error can't find the container with id 29b436635de40aefc0f240ff79ac016828687067bb9ab82a73e3b2b541bad5ed Jul 21 04:16:19.325692 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:19.325662 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:19.381949 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:19.381915 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-964f5b447-8hjqx" event={"ID":"e6412c0e-d008-46d9-9296-cfdc4ebe666e","Type":"ContainerStarted","Data":"1886ae46ff7d92ab301f932c66247ed9a19dc2d33ecc86198755998e8aca1b06"} Jul 21 04:16:19.382136 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:19.381965 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-964f5b447-8hjqx" event={"ID":"e6412c0e-d008-46d9-9296-cfdc4ebe666e","Type":"ContainerStarted","Data":"29b436635de40aefc0f240ff79ac016828687067bb9ab82a73e3b2b541bad5ed"} Jul 21 04:16:19.403866 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:19.403807 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-964f5b447-8hjqx" podStartSLOduration=1.403786512 podStartE2EDuration="1.403786512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:16:19.399901739 +0000 UTC m=+424.799234509" watchObservedRunningTime="2026-07-21 04:16:19.403786512 +0000 UTC m=+424.803119273" Jul 21 04:16:21.394813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:21.394779 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" event={"ID":"a53b8d99-c7a1-4830-86e4-e87a91283cdb","Type":"ContainerStarted","Data":"b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750"} Jul 21 04:16:21.395199 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:21.394885 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:21.417393 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:21.417288 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" podStartSLOduration=0.843534169 podStartE2EDuration="5.417276241s" totalImagesPullingTime="4.573742072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:16:21.41398361 +0000 UTC m=+426.813316368" watchObservedRunningTime="2026-07-21 04:16:21.417276241 +0000 UTC m=+426.816608998" Jul 21 04:16:23.988049 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:23.988014 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n"] Jul 21 04:16:23.991830 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:23.991811 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:23.994615 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:23.994593 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 21 04:16:23.994722 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:23.994594 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wx5hb\"" Jul 21 04:16:23.994722 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:23.994592 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 21 04:16:24.002731 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.002711 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n"] Jul 21 04:16:24.146278 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.146244 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/248149d4-7aed-4d5f-98e0-102719e9f638-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.146278 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.146281 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzmss\" (UniqueName: \"kubernetes.io/projected/248149d4-7aed-4d5f-98e0-102719e9f638-kube-api-access-lzmss\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.146528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.146324 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/248149d4-7aed-4d5f-98e0-102719e9f638-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.247181 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.247100 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/248149d4-7aed-4d5f-98e0-102719e9f638-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.247181 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.247169 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/248149d4-7aed-4d5f-98e0-102719e9f638-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.247434 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.247189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzmss\" (UniqueName: \"kubernetes.io/projected/248149d4-7aed-4d5f-98e0-102719e9f638-kube-api-access-lzmss\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.247434 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:24.247266 2567 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 21 04:16:24.247434 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:24.247382 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/248149d4-7aed-4d5f-98e0-102719e9f638-plugin-serving-cert podName:248149d4-7aed-4d5f-98e0-102719e9f638 nodeName:}" failed. No retries permitted until 2026-07-21 04:16:24.747332601 +0000 UTC m=+430.146665338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/248149d4-7aed-4d5f-98e0-102719e9f638-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-88r7n" (UID: "248149d4-7aed-4d5f-98e0-102719e9f638") : secret "plugin-serving-cert" not found Jul 21 04:16:24.247805 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.247787 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/248149d4-7aed-4d5f-98e0-102719e9f638-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.257895 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.257862 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzmss\" (UniqueName: \"kubernetes.io/projected/248149d4-7aed-4d5f-98e0-102719e9f638-kube-api-access-lzmss\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.355023 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.354997 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zzxt" Jul 21 04:16:24.751458 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.751428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/248149d4-7aed-4d5f-98e0-102719e9f638-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.753800 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.753770 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/248149d4-7aed-4d5f-98e0-102719e9f638-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-88r7n\" (UID: \"248149d4-7aed-4d5f-98e0-102719e9f638\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:24.900969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:24.900938 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" Jul 21 04:16:25.035993 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:25.035971 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n"] Jul 21 04:16:25.037608 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:16:25.037575 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod248149d4_7aed_4d5f_98e0_102719e9f638.slice/crio-4e6ed624d6cc0dde9fab8bb87832abf4761a9dc36f34e1bec98ea491fd936066 WatchSource:0}: Error finding container 4e6ed624d6cc0dde9fab8bb87832abf4761a9dc36f34e1bec98ea491fd936066: Status 404 returned error can't find the container with id 4e6ed624d6cc0dde9fab8bb87832abf4761a9dc36f34e1bec98ea491fd936066 Jul 21 04:16:25.417270 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:25.417186 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" event={"ID":"248149d4-7aed-4d5f-98e0-102719e9f638","Type":"ContainerStarted","Data":"4e6ed624d6cc0dde9fab8bb87832abf4761a9dc36f34e1bec98ea491fd936066"} Jul 21 04:16:28.464066 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:28.464026 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:28.464562 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:28.464089 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:28.470096 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:28.470071 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:29.451319 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:29.451290 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-964f5b447-8hjqx" Jul 21 04:16:29.503850 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:29.503756 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b55685479-frshb"] Jul 21 04:16:32.403160 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:32.403125 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:33.370690 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.370656 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt"] Jul 21 04:16:33.395502 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.395466 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt"] Jul 21 04:16:33.395685 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.395533 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:33.536769 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.536730 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzmqx\" (UniqueName: \"kubernetes.io/projected/0454e071-6bff-4df0-b429-d2832c191ced-kube-api-access-wzmqx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" (UID: \"0454e071-6bff-4df0-b429-d2832c191ced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:33.536769 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.536775 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0454e071-6bff-4df0-b429-d2832c191ced-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" (UID: \"0454e071-6bff-4df0-b429-d2832c191ced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:33.638142 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.638057 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzmqx\" (UniqueName: \"kubernetes.io/projected/0454e071-6bff-4df0-b429-d2832c191ced-kube-api-access-wzmqx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" (UID: \"0454e071-6bff-4df0-b429-d2832c191ced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:33.638142 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.638115 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0454e071-6bff-4df0-b429-d2832c191ced-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" (UID: \"0454e071-6bff-4df0-b429-d2832c191ced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:33.638594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.638562 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0454e071-6bff-4df0-b429-d2832c191ced-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" (UID: \"0454e071-6bff-4df0-b429-d2832c191ced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:33.647555 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.647528 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzmqx\" (UniqueName: \"kubernetes.io/projected/0454e071-6bff-4df0-b429-d2832c191ced-kube-api-access-wzmqx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" (UID: \"0454e071-6bff-4df0-b429-d2832c191ced\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:33.709439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:33.709393 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:34.024131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.024088 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt"] Jul 21 04:16:34.027005 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.026976 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt"] Jul 21 04:16:34.039043 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.039005 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg"] Jul 21 04:16:34.040131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.039498 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" containerName="manager" containerID="cri-o://b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750" gracePeriod=2 Jul 21 04:16:34.049410 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.045215 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg"] Jul 21 04:16:34.049585 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.049569 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt"] Jul 21 04:16:34.050175 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.050156 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" containerName="manager" Jul 21 04:16:34.050294 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.050280 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" containerName="manager" Jul 21 04:16:34.050774 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.050756 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" containerName="manager" Jul 21 04:16:34.076057 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:34.074864 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:34.076057 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.075281 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts"] Jul 21 04:16:34.076057 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.075328 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f"] Jul 21 04:16:34.076057 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.075616 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" podUID="2e92db26-372e-42cf-b341-98492407bd24" containerName="manager" containerID="cri-o://20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1" gracePeriod=2 Jul 21 04:16:34.076379 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.076193 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:34.080989 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:34.079313 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:34.082224 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:34.082034 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dvkts\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="2e92db26-372e-42cf-b341-98492407bd24" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:34.084820 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:34.084470 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dvkts\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="2e92db26-372e-42cf-b341-98492407bd24" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:34.086025 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.086008 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts"] Jul 21 04:16:34.086212 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.086189 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt"] Jul 21 04:16:34.086422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.086111 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:34.086683 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.086662 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f"] Jul 21 04:16:34.109355 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:34.109295 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:34.111530 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:34.111481 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dvkts\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="2e92db26-372e-42cf-b341-98492407bd24" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:34.114362 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:34.113823 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:34.250996 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.250955 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wckq\" (UniqueName: \"kubernetes.io/projected/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-kube-api-access-8wckq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4l8qt\" (UID: \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:34.251186 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.251002 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8xmn\" (UniqueName: \"kubernetes.io/projected/521e40b0-bad8-4068-9c81-468169d18a1b-kube-api-access-c8xmn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v5k5f\" (UID: \"521e40b0-bad8-4068-9c81-468169d18a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:34.251186 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.251033 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/521e40b0-bad8-4068-9c81-468169d18a1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v5k5f\" (UID: \"521e40b0-bad8-4068-9c81-468169d18a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:34.251186 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.251134 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4l8qt\" (UID: \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:34.352034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.351944 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4l8qt\" (UID: \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:34.352203 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.352056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wckq\" (UniqueName: \"kubernetes.io/projected/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-kube-api-access-8wckq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4l8qt\" (UID: \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:34.352203 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.352081 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8xmn\" (UniqueName: \"kubernetes.io/projected/521e40b0-bad8-4068-9c81-468169d18a1b-kube-api-access-c8xmn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v5k5f\" (UID: \"521e40b0-bad8-4068-9c81-468169d18a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:34.352203 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.352105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/521e40b0-bad8-4068-9c81-468169d18a1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v5k5f\" (UID: \"521e40b0-bad8-4068-9c81-468169d18a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:34.352564 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.352526 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/521e40b0-bad8-4068-9c81-468169d18a1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v5k5f\" (UID: \"521e40b0-bad8-4068-9c81-468169d18a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:34.352804 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.352772 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4l8qt\" (UID: \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:34.362226 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.362198 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wckq\" (UniqueName: \"kubernetes.io/projected/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-kube-api-access-8wckq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4l8qt\" (UID: \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:34.365021 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.364995 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8xmn\" (UniqueName: \"kubernetes.io/projected/521e40b0-bad8-4068-9c81-468169d18a1b-kube-api-access-c8xmn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v5k5f\" (UID: \"521e40b0-bad8-4068-9c81-468169d18a1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:34.467648 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.467616 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:34.475191 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:34.475164 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:35.265904 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:35.265862 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dvkts\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="2e92db26-372e-42cf-b341-98492407bd24" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:35.268115 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:35.268084 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:45.763951 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.763925 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:45.765704 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.765680 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt"] Jul 21 04:16:45.766134 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:45.766109 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:45.767394 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:16:45.767355 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ecd0af_4704_48b8_80bc_8f4a37bdb81d.slice/crio-52aca52ccb16b6c17835c86b93119db871afd82d1b52549ce418803252dba48d WatchSource:0}: Error finding container 52aca52ccb16b6c17835c86b93119db871afd82d1b52549ce418803252dba48d: Status 404 returned error can't find the container with id 52aca52ccb16b6c17835c86b93119db871afd82d1b52549ce418803252dba48d Jul 21 04:16:45.769482 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.769362 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:45.771740 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:45.771711 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dvkts\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="2e92db26-372e-42cf-b341-98492407bd24" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:45.773684 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:45.773659 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:45.793580 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.793559 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f"] Jul 21 04:16:45.796462 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:16:45.796438 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521e40b0_bad8_4068_9c81_468169d18a1b.slice/crio-f2695419280bbd7c3e45a87a549ff838cd547acdb0328f31067924c94ea9c593 WatchSource:0}: Error finding container f2695419280bbd7c3e45a87a549ff838cd547acdb0328f31067924c94ea9c593: Status 404 returned error can't find the container with id f2695419280bbd7c3e45a87a549ff838cd547acdb0328f31067924c94ea9c593 Jul 21 04:16:45.859027 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.859001 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwljz\" (UniqueName: \"kubernetes.io/projected/2e92db26-372e-42cf-b341-98492407bd24-kube-api-access-jwljz\") pod \"2e92db26-372e-42cf-b341-98492407bd24\" (UID: \"2e92db26-372e-42cf-b341-98492407bd24\") " Jul 21 04:16:45.859141 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.859057 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53b8d99-c7a1-4830-86e4-e87a91283cdb-extensions-socket-volume\") pod \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\" (UID: \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\") " Jul 21 04:16:45.859212 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.859147 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h6f6\" (UniqueName: \"kubernetes.io/projected/a53b8d99-c7a1-4830-86e4-e87a91283cdb-kube-api-access-8h6f6\") pod \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\" (UID: \"a53b8d99-c7a1-4830-86e4-e87a91283cdb\") " Jul 21 04:16:45.859565 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.859541 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b8d99-c7a1-4830-86e4-e87a91283cdb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a53b8d99-c7a1-4830-86e4-e87a91283cdb" (UID: "a53b8d99-c7a1-4830-86e4-e87a91283cdb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:16:45.860907 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.860883 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e92db26-372e-42cf-b341-98492407bd24-kube-api-access-jwljz" (OuterVolumeSpecName: "kube-api-access-jwljz") pod "2e92db26-372e-42cf-b341-98492407bd24" (UID: "2e92db26-372e-42cf-b341-98492407bd24"). InnerVolumeSpecName "kube-api-access-jwljz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:16:45.861005 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.860987 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53b8d99-c7a1-4830-86e4-e87a91283cdb-kube-api-access-8h6f6" (OuterVolumeSpecName: "kube-api-access-8h6f6") pod "a53b8d99-c7a1-4830-86e4-e87a91283cdb" (UID: "a53b8d99-c7a1-4830-86e4-e87a91283cdb"). InnerVolumeSpecName "kube-api-access-8h6f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:16:45.960472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.960443 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h6f6\" (UniqueName: \"kubernetes.io/projected/a53b8d99-c7a1-4830-86e4-e87a91283cdb-kube-api-access-8h6f6\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:45.960472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.960468 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwljz\" (UniqueName: \"kubernetes.io/projected/2e92db26-372e-42cf-b341-98492407bd24-kube-api-access-jwljz\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:45.960472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:45.960479 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a53b8d99-c7a1-4830-86e4-e87a91283cdb-extensions-socket-volume\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:46.552272 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.552236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" event={"ID":"521e40b0-bad8-4068-9c81-468169d18a1b","Type":"ContainerStarted","Data":"d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843"} Jul 21 04:16:46.552467 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.552289 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" event={"ID":"521e40b0-bad8-4068-9c81-468169d18a1b","Type":"ContainerStarted","Data":"f2695419280bbd7c3e45a87a549ff838cd547acdb0328f31067924c94ea9c593"} Jul 21 04:16:46.552467 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.552381 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:46.554020 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.553997 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" event={"ID":"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d","Type":"ContainerStarted","Data":"b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64"} Jul 21 04:16:46.554145 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.554033 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" event={"ID":"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d","Type":"ContainerStarted","Data":"52aca52ccb16b6c17835c86b93119db871afd82d1b52549ce418803252dba48d"} Jul 21 04:16:46.554145 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.554081 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:46.554605 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:46.554580 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dvkts\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="2e92db26-372e-42cf-b341-98492407bd24" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:46.555296 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.555278 2567 generic.go:356] "Generic (PLEG): container finished" podID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" containerID="b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750" exitCode=0 Jul 21 04:16:46.555397 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.555311 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:46.555397 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.555362 2567 scope.go:117] "RemoveContainer" containerID="b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750" Jul 21 04:16:46.556944 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.556921 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" event={"ID":"248149d4-7aed-4d5f-98e0-102719e9f638","Type":"ContainerStarted","Data":"c6b0c2890d56167bb5900741f4604cc9f15c367457e4c18b98aa08595e87a5ce"} Jul 21 04:16:46.557056 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:46.557016 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:46.558301 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.558267 2567 generic.go:356] "Generic (PLEG): container finished" podID="2e92db26-372e-42cf-b341-98492407bd24" containerID="20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1" exitCode=0 Jul 21 04:16:46.558389 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.558298 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:46.565652 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.565636 2567 scope.go:117] "RemoveContainer" containerID="b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750" Jul 21 04:16:46.565892 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:46.565875 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750\": container with ID starting with b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750 not found: ID does not exist" containerID="b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750" Jul 21 04:16:46.565942 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.565897 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750"} err="failed to get container status \"b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750\": rpc error: code = NotFound desc = could not find container \"b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750\": container with ID starting with b08a6b5e3361ad7df9c25c649bb3f94236c809c5092fcda09e28e43e11287750 not found: ID does not exist" Jul 21 04:16:46.565942 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.565910 2567 scope.go:117] "RemoveContainer" containerID="20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1" Jul 21 04:16:46.574018 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.574003 2567 scope.go:117] "RemoveContainer" containerID="20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1" Jul 21 04:16:46.574246 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:46.574231 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1\": container with ID starting with 20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1 not found: ID does not exist" containerID="20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1" Jul 21 04:16:46.574305 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.574256 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1"} err="failed to get container status \"20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1\": rpc error: code = NotFound desc = could not find container \"20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1\": container with ID starting with 20281ef9743ebf3f4e40892c3815d3c5e18988871ac289afe72070cb22a1fde1 not found: ID does not exist" Jul 21 04:16:46.575933 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.575890 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" podStartSLOduration=12.575874471 podStartE2EDuration="12.575874471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:16:46.574882046 +0000 UTC m=+451.974214829" watchObservedRunningTime="2026-07-21 04:16:46.575874471 +0000 UTC m=+451.975207228" Jul 21 04:16:46.591588 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.591552 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-88r7n" podStartSLOduration=2.944707549 podStartE2EDuration="23.591541488s" totalImagesPullingTime="20.646833939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:16:46.590160838 +0000 UTC m=+451.989493595" watchObservedRunningTime="2026-07-21 04:16:46.591541488 +0000 UTC m=+451.990874263" Jul 21 04:16:46.592644 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:46.592623 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-dvkts\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="2e92db26-372e-42cf-b341-98492407bd24" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dvkts" Jul 21 04:16:46.620303 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:46.618236 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-stfcg\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-stfcg" Jul 21 04:16:46.621804 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:46.621754 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" podStartSLOduration=12.621732376 podStartE2EDuration="12.621732376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:16:46.611214583 +0000 UTC m=+452.010547343" watchObservedRunningTime="2026-07-21 04:16:46.621732376 +0000 UTC m=+452.021065135" Jul 21 04:16:47.266443 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:47.266411 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e92db26-372e-42cf-b341-98492407bd24" path="/var/lib/kubelet/pods/2e92db26-372e-42cf-b341-98492407bd24/volumes" Jul 21 04:16:47.266840 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:47.266769 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53b8d99-c7a1-4830-86e4-e87a91283cdb" path="/var/lib/kubelet/pods/a53b8d99-c7a1-4830-86e4-e87a91283cdb/volumes" Jul 21 04:16:48.130811 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:48.130769 2567 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-hxqmt_kuadrant-system_0454e071-6bff-4df0-b429-d2832c191ced_0(9e081b9c9e3321c14a593fc64557d55f735df4251d7f3ba8cd27fb382ad01f72): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-hxqmt to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 21 04:16:48.130930 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:48.130834 2567 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-hxqmt_kuadrant-system_0454e071-6bff-4df0-b429-d2832c191ced_0(9e081b9c9e3321c14a593fc64557d55f735df4251d7f3ba8cd27fb382ad01f72): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-hxqmt to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:48.570827 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:48.570797 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:48.573532 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:48.573498 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="0454e071-6bff-4df0-b429-d2832c191ced" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:48.579217 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:48.579195 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:48.581715 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:48.581692 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="0454e071-6bff-4df0-b429-d2832c191ced" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:48.679780 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:48.679760 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzmqx\" (UniqueName: \"kubernetes.io/projected/0454e071-6bff-4df0-b429-d2832c191ced-kube-api-access-wzmqx\") pod \"0454e071-6bff-4df0-b429-d2832c191ced\" (UID: \"0454e071-6bff-4df0-b429-d2832c191ced\") " Jul 21 04:16:48.679901 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:48.679810 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0454e071-6bff-4df0-b429-d2832c191ced-extensions-socket-volume\") pod \"0454e071-6bff-4df0-b429-d2832c191ced\" (UID: \"0454e071-6bff-4df0-b429-d2832c191ced\") " Jul 21 04:16:48.680118 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:48.680094 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0454e071-6bff-4df0-b429-d2832c191ced-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0454e071-6bff-4df0-b429-d2832c191ced" (UID: "0454e071-6bff-4df0-b429-d2832c191ced"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:16:48.681970 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:48.681945 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0454e071-6bff-4df0-b429-d2832c191ced-kube-api-access-wzmqx" (OuterVolumeSpecName: "kube-api-access-wzmqx") pod "0454e071-6bff-4df0-b429-d2832c191ced" (UID: "0454e071-6bff-4df0-b429-d2832c191ced"). InnerVolumeSpecName "kube-api-access-wzmqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:16:48.780439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:48.780416 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0454e071-6bff-4df0-b429-d2832c191ced-extensions-socket-volume\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:48.780439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:48.780437 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzmqx\" (UniqueName: \"kubernetes.io/projected/0454e071-6bff-4df0-b429-d2832c191ced-kube-api-access-wzmqx\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:49.267198 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:49.267168 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0454e071-6bff-4df0-b429-d2832c191ced" path="/var/lib/kubelet/pods/0454e071-6bff-4df0-b429-d2832c191ced/volumes" Jul 21 04:16:49.576123 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:49.576034 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:49.578498 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:49.578466 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="0454e071-6bff-4df0-b429-d2832c191ced" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:49.580623 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:49.580602 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-hxqmt\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="0454e071-6bff-4df0-b429-d2832c191ced" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hxqmt" Jul 21 04:16:54.534244 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.534209 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b55685479-frshb" podUID="8919a027-ee74-4eb4-9157-52c71c5e40ce" containerName="console" containerID="cri-o://fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9" gracePeriod=15 Jul 21 04:16:54.822610 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.822583 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b55685479-frshb" Jul 21 04:16:54.935447 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.935413 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-service-ca\") pod \"8919a027-ee74-4eb4-9157-52c71c5e40ce\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " Jul 21 04:16:54.935447 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.935450 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-config\") pod \"8919a027-ee74-4eb4-9157-52c71c5e40ce\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " Jul 21 04:16:54.935675 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.935501 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-serving-cert\") pod \"8919a027-ee74-4eb4-9157-52c71c5e40ce\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " Jul 21 04:16:54.935675 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.935529 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-oauth-serving-cert\") pod \"8919a027-ee74-4eb4-9157-52c71c5e40ce\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " Jul 21 04:16:54.935675 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.935555 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxvm4\" (UniqueName: \"kubernetes.io/projected/8919a027-ee74-4eb4-9157-52c71c5e40ce-kube-api-access-cxvm4\") pod \"8919a027-ee74-4eb4-9157-52c71c5e40ce\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " Jul 21 04:16:54.935675 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.935584 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-trusted-ca-bundle\") pod \"8919a027-ee74-4eb4-9157-52c71c5e40ce\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " Jul 21 04:16:54.935675 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.935617 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-oauth-config\") pod \"8919a027-ee74-4eb4-9157-52c71c5e40ce\" (UID: \"8919a027-ee74-4eb4-9157-52c71c5e40ce\") " Jul 21 04:16:54.935993 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.935964 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-service-ca" (OuterVolumeSpecName: "service-ca") pod "8919a027-ee74-4eb4-9157-52c71c5e40ce" (UID: "8919a027-ee74-4eb4-9157-52c71c5e40ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:16:54.936135 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.936101 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-config" (OuterVolumeSpecName: "console-config") pod "8919a027-ee74-4eb4-9157-52c71c5e40ce" (UID: "8919a027-ee74-4eb4-9157-52c71c5e40ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:16:54.936259 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.936151 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8919a027-ee74-4eb4-9157-52c71c5e40ce" (UID: "8919a027-ee74-4eb4-9157-52c71c5e40ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:16:54.936259 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.936162 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8919a027-ee74-4eb4-9157-52c71c5e40ce" (UID: "8919a027-ee74-4eb4-9157-52c71c5e40ce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:16:54.937870 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.937847 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8919a027-ee74-4eb4-9157-52c71c5e40ce" (UID: "8919a027-ee74-4eb4-9157-52c71c5e40ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:16:54.937957 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.937865 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8919a027-ee74-4eb4-9157-52c71c5e40ce-kube-api-access-cxvm4" (OuterVolumeSpecName: "kube-api-access-cxvm4") pod "8919a027-ee74-4eb4-9157-52c71c5e40ce" (UID: "8919a027-ee74-4eb4-9157-52c71c5e40ce"). InnerVolumeSpecName "kube-api-access-cxvm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:16:54.938009 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:54.937982 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8919a027-ee74-4eb4-9157-52c71c5e40ce" (UID: "8919a027-ee74-4eb4-9157-52c71c5e40ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:16:55.036803 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.036771 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-service-ca\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:55.036803 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.036797 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:55.036803 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.036808 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-serving-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:55.037025 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.036816 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-oauth-serving-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:55.037025 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.036827 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxvm4\" (UniqueName: \"kubernetes.io/projected/8919a027-ee74-4eb4-9157-52c71c5e40ce-kube-api-access-cxvm4\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:55.037025 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.036836 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8919a027-ee74-4eb4-9157-52c71c5e40ce-trusted-ca-bundle\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:55.037025 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.036844 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8919a027-ee74-4eb4-9157-52c71c5e40ce-console-oauth-config\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:55.608225 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.608145 2567 generic.go:356] "Generic (PLEG): container finished" podID="8919a027-ee74-4eb4-9157-52c71c5e40ce" containerID="fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9" exitCode=0 Jul 21 04:16:55.608225 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.608191 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b55685479-frshb" Jul 21 04:16:55.608709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.608227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b55685479-frshb" event={"ID":"8919a027-ee74-4eb4-9157-52c71c5e40ce","Type":"ContainerDied","Data":"fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9"} Jul 21 04:16:55.608709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.608278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b55685479-frshb" event={"ID":"8919a027-ee74-4eb4-9157-52c71c5e40ce","Type":"ContainerDied","Data":"04d27b1956191601e814006fcbda898c2617a1c80514f9d983cd5226dcbba825"} Jul 21 04:16:55.608709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.608304 2567 scope.go:117] "RemoveContainer" containerID="fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9" Jul 21 04:16:55.617149 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.617120 2567 scope.go:117] "RemoveContainer" containerID="fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9" Jul 21 04:16:55.617430 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:55.617408 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9\": container with ID starting with fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9 not found: ID does not exist" containerID="fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9" Jul 21 04:16:55.617488 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.617447 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9"} err="failed to get container status \"fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9\": rpc error: code = NotFound desc = could not find container \"fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9\": container with ID starting with fd12762aa9c3cedc6c1aed07384cd71a4406a2a447fb664738cf342c50e8a1e9 not found: ID does not exist" Jul 21 04:16:55.635880 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.635855 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b55685479-frshb"] Jul 21 04:16:55.642216 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:55.642196 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b55685479-frshb"] Jul 21 04:16:57.265991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.265957 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8919a027-ee74-4eb4-9157-52c71c5e40ce" path="/var/lib/kubelet/pods/8919a027-ee74-4eb4-9157-52c71c5e40ce/volumes" Jul 21 04:16:57.567535 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.567456 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:57.567678 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.567541 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:16:57.637150 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.637117 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f"] Jul 21 04:16:57.637386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.637293 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" podUID="521e40b0-bad8-4068-9c81-468169d18a1b" containerName="manager" containerID="cri-o://d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843" gracePeriod=10 Jul 21 04:16:57.844185 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.844099 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp"] Jul 21 04:16:57.844627 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.844599 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e92db26-372e-42cf-b341-98492407bd24" containerName="manager" Jul 21 04:16:57.844627 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.844623 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e92db26-372e-42cf-b341-98492407bd24" containerName="manager" Jul 21 04:16:57.844844 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.844665 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8919a027-ee74-4eb4-9157-52c71c5e40ce" containerName="console" Jul 21 04:16:57.844844 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.844673 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8919a027-ee74-4eb4-9157-52c71c5e40ce" containerName="console" Jul 21 04:16:57.844844 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.844787 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8919a027-ee74-4eb4-9157-52c71c5e40ce" containerName="console" Jul 21 04:16:57.844844 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.844805 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e92db26-372e-42cf-b341-98492407bd24" containerName="manager" Jul 21 04:16:57.846795 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.846777 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:16:57.860454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.860433 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp"] Jul 21 04:16:57.888029 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.888007 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:57.962925 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.962892 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/521e40b0-bad8-4068-9c81-468169d18a1b-extensions-socket-volume\") pod \"521e40b0-bad8-4068-9c81-468169d18a1b\" (UID: \"521e40b0-bad8-4068-9c81-468169d18a1b\") " Jul 21 04:16:57.963075 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.962939 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8xmn\" (UniqueName: \"kubernetes.io/projected/521e40b0-bad8-4068-9c81-468169d18a1b-kube-api-access-c8xmn\") pod \"521e40b0-bad8-4068-9c81-468169d18a1b\" (UID: \"521e40b0-bad8-4068-9c81-468169d18a1b\") " Jul 21 04:16:57.963075 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.963028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/179c9a09-3ff7-431f-9305-5fb4bbe5f128-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5d7jp\" (UID: \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:16:57.963159 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.963088 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-222s2\" (UniqueName: \"kubernetes.io/projected/179c9a09-3ff7-431f-9305-5fb4bbe5f128-kube-api-access-222s2\") pod \"kuadrant-operator-controller-manager-86878c769f-5d7jp\" (UID: \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:16:57.963267 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.963244 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/521e40b0-bad8-4068-9c81-468169d18a1b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "521e40b0-bad8-4068-9c81-468169d18a1b" (UID: "521e40b0-bad8-4068-9c81-468169d18a1b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:16:57.964878 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:57.964855 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521e40b0-bad8-4068-9c81-468169d18a1b-kube-api-access-c8xmn" (OuterVolumeSpecName: "kube-api-access-c8xmn") pod "521e40b0-bad8-4068-9c81-468169d18a1b" (UID: "521e40b0-bad8-4068-9c81-468169d18a1b"). InnerVolumeSpecName "kube-api-access-c8xmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:16:58.063813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.063772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/179c9a09-3ff7-431f-9305-5fb4bbe5f128-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5d7jp\" (UID: \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:16:58.064007 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.063855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-222s2\" (UniqueName: \"kubernetes.io/projected/179c9a09-3ff7-431f-9305-5fb4bbe5f128-kube-api-access-222s2\") pod \"kuadrant-operator-controller-manager-86878c769f-5d7jp\" (UID: \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:16:58.064007 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.063911 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/521e40b0-bad8-4068-9c81-468169d18a1b-extensions-socket-volume\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:58.064007 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.063925 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8xmn\" (UniqueName: \"kubernetes.io/projected/521e40b0-bad8-4068-9c81-468169d18a1b-kube-api-access-c8xmn\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:16:58.064164 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.064149 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/179c9a09-3ff7-431f-9305-5fb4bbe5f128-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5d7jp\" (UID: \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:16:58.078820 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.078793 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-222s2\" (UniqueName: \"kubernetes.io/projected/179c9a09-3ff7-431f-9305-5fb4bbe5f128-kube-api-access-222s2\") pod \"kuadrant-operator-controller-manager-86878c769f-5d7jp\" (UID: \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:16:58.156948 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.156872 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:16:58.292178 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.292150 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp"] Jul 21 04:16:58.294010 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:16:58.293985 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179c9a09_3ff7_431f_9305_5fb4bbe5f128.slice/crio-920d150f69d285177a2af82c88203713e8982cd0d628e6113813e6c9b9c9db9b WatchSource:0}: Error finding container 920d150f69d285177a2af82c88203713e8982cd0d628e6113813e6c9b9c9db9b: Status 404 returned error can't find the container with id 920d150f69d285177a2af82c88203713e8982cd0d628e6113813e6c9b9c9db9b Jul 21 04:16:58.630164 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.630107 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" event={"ID":"179c9a09-3ff7-431f-9305-5fb4bbe5f128","Type":"ContainerStarted","Data":"1a7262233f429f31db669aff9b12beb5d09f30f7febc79fc63f0684fddcd4069"} Jul 21 04:16:58.630408 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.630188 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:16:58.630408 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.630213 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" event={"ID":"179c9a09-3ff7-431f-9305-5fb4bbe5f128","Type":"ContainerStarted","Data":"920d150f69d285177a2af82c88203713e8982cd0d628e6113813e6c9b9c9db9b"} Jul 21 04:16:58.631430 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.631406 2567 generic.go:356] "Generic (PLEG): container finished" podID="521e40b0-bad8-4068-9c81-468169d18a1b" containerID="d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843" exitCode=0 Jul 21 04:16:58.631538 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.631454 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" event={"ID":"521e40b0-bad8-4068-9c81-468169d18a1b","Type":"ContainerDied","Data":"d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843"} Jul 21 04:16:58.631538 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.631457 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" Jul 21 04:16:58.631538 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.631490 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f" event={"ID":"521e40b0-bad8-4068-9c81-468169d18a1b","Type":"ContainerDied","Data":"f2695419280bbd7c3e45a87a549ff838cd547acdb0328f31067924c94ea9c593"} Jul 21 04:16:58.631538 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.631510 2567 scope.go:117] "RemoveContainer" containerID="d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843" Jul 21 04:16:58.640624 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.640440 2567 scope.go:117] "RemoveContainer" containerID="d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843" Jul 21 04:16:58.640699 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:16:58.640679 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843\": container with ID starting with d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843 not found: ID does not exist" containerID="d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843" Jul 21 04:16:58.640742 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.640706 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843"} err="failed to get container status \"d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843\": rpc error: code = NotFound desc = could not find container \"d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843\": container with ID starting with d222a01e743d5e776bf11b162c57af2745be90e626eddaf6e9470ec8055f5843 not found: ID does not exist" Jul 21 04:16:58.651029 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.650988 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" podStartSLOduration=1.650976001 podStartE2EDuration="1.650976001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:16:58.649489588 +0000 UTC m=+464.048822346" watchObservedRunningTime="2026-07-21 04:16:58.650976001 +0000 UTC m=+464.050308758" Jul 21 04:16:58.665194 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.665171 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f"] Jul 21 04:16:58.669291 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:58.669268 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v5k5f"] Jul 21 04:16:59.266501 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:16:59.266471 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="521e40b0-bad8-4068-9c81-468169d18a1b" path="/var/lib/kubelet/pods/521e40b0-bad8-4068-9c81-468169d18a1b/volumes" Jul 21 04:17:09.639988 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:09.639950 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:17:09.690231 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:09.690191 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt"] Jul 21 04:17:09.690560 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:09.690474 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" podUID="d8ecd0af-4704-48b8-80bc-8f4a37bdb81d" containerName="manager" containerID="cri-o://b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64" gracePeriod=10 Jul 21 04:17:09.957647 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:09.957623 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:17:10.063522 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.063488 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-extensions-socket-volume\") pod \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\" (UID: \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\") " Jul 21 04:17:10.063688 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.063590 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wckq\" (UniqueName: \"kubernetes.io/projected/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-kube-api-access-8wckq\") pod \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\" (UID: \"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d\") " Jul 21 04:17:10.063840 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.063818 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d8ecd0af-4704-48b8-80bc-8f4a37bdb81d" (UID: "d8ecd0af-4704-48b8-80bc-8f4a37bdb81d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:17:10.065548 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.065518 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-kube-api-access-8wckq" (OuterVolumeSpecName: "kube-api-access-8wckq") pod "d8ecd0af-4704-48b8-80bc-8f4a37bdb81d" (UID: "d8ecd0af-4704-48b8-80bc-8f4a37bdb81d"). InnerVolumeSpecName "kube-api-access-8wckq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:17:10.164822 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.164796 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-extensions-socket-volume\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:17:10.164822 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.164821 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wckq\" (UniqueName: \"kubernetes.io/projected/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d-kube-api-access-8wckq\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:17:10.702418 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.702380 2567 generic.go:356] "Generic (PLEG): container finished" podID="d8ecd0af-4704-48b8-80bc-8f4a37bdb81d" containerID="b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64" exitCode=0 Jul 21 04:17:10.702856 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.702434 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" Jul 21 04:17:10.702856 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.702475 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" event={"ID":"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d","Type":"ContainerDied","Data":"b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64"} Jul 21 04:17:10.702856 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.702530 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt" event={"ID":"d8ecd0af-4704-48b8-80bc-8f4a37bdb81d","Type":"ContainerDied","Data":"52aca52ccb16b6c17835c86b93119db871afd82d1b52549ce418803252dba48d"} Jul 21 04:17:10.702856 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.702552 2567 scope.go:117] "RemoveContainer" containerID="b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64" Jul 21 04:17:10.711650 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.711634 2567 scope.go:117] "RemoveContainer" containerID="b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64" Jul 21 04:17:10.711896 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:17:10.711875 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64\": container with ID starting with b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64 not found: ID does not exist" containerID="b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64" Jul 21 04:17:10.712011 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.711898 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64"} err="failed to get container status \"b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64\": rpc error: code = NotFound desc = could not find container \"b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64\": container with ID starting with b4bc403e4798fefe92a769a38e6766b6f93dacce7323c4d2d7863ed89fbd4a64 not found: ID does not exist" Jul 21 04:17:10.728954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.728931 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt"] Jul 21 04:17:10.733239 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:10.733217 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4l8qt"] Jul 21 04:17:11.266623 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:11.266595 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8ecd0af-4704-48b8-80bc-8f4a37bdb81d" path="/var/lib/kubelet/pods/d8ecd0af-4704-48b8-80bc-8f4a37bdb81d/volumes" Jul 21 04:17:13.972005 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:13.971965 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k"] Jul 21 04:17:13.972564 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:13.972542 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="521e40b0-bad8-4068-9c81-468169d18a1b" containerName="manager" Jul 21 04:17:13.972641 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:13.972567 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="521e40b0-bad8-4068-9c81-468169d18a1b" containerName="manager" Jul 21 04:17:13.972641 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:13.972584 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8ecd0af-4704-48b8-80bc-8f4a37bdb81d" containerName="manager" Jul 21 04:17:13.972641 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:13.972592 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8ecd0af-4704-48b8-80bc-8f4a37bdb81d" containerName="manager" Jul 21 04:17:13.972848 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:13.972704 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8ecd0af-4704-48b8-80bc-8f4a37bdb81d" containerName="manager" Jul 21 04:17:13.972848 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:13.972719 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="521e40b0-bad8-4068-9c81-468169d18a1b" containerName="manager" Jul 21 04:17:14.013423 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.013388 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k"] Jul 21 04:17:14.013571 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.013495 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.016021 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.015995 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 04:17:14.016021 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.015995 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q87rv\"" Jul 21 04:17:14.098308 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.098282 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.098457 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.098319 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.098457 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.098375 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.098457 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.098407 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hbr5\" (UniqueName: \"kubernetes.io/projected/0218007d-5768-4855-ac24-f35d5daf2d66-kube-api-access-6hbr5\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.098457 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.098430 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0218007d-5768-4855-ac24-f35d5daf2d66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.098594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.098468 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0218007d-5768-4855-ac24-f35d5daf2d66-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.098594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.098504 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0218007d-5768-4855-ac24-f35d5daf2d66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.098594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.098562 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.098594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.098588 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199166 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199133 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199166 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199166 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199197 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199222 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199247 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199266 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hbr5\" (UniqueName: \"kubernetes.io/projected/0218007d-5768-4855-ac24-f35d5daf2d66-kube-api-access-6hbr5\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199284 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0218007d-5768-4855-ac24-f35d5daf2d66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199303 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0218007d-5768-4855-ac24-f35d5daf2d66-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199368 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0218007d-5768-4855-ac24-f35d5daf2d66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199735 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199581 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199735 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199634 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.199813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.199798 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.200103 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.200082 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0218007d-5768-4855-ac24-f35d5daf2d66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.201738 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.201711 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0218007d-5768-4855-ac24-f35d5daf2d66-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.201827 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.201800 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0218007d-5768-4855-ac24-f35d5daf2d66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.208633 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.208607 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hbr5\" (UniqueName: \"kubernetes.io/projected/0218007d-5768-4855-ac24-f35d5daf2d66-kube-api-access-6hbr5\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.208934 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.208917 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0218007d-5768-4855-ac24-f35d5daf2d66-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rdt7k\" (UID: \"0218007d-5768-4855-ac24-f35d5daf2d66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.322971 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.322947 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:14.464834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.464793 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k"] Jul 21 04:17:14.470587 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:17:14.470556 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0218007d_5768_4855_ac24_f35d5daf2d66.slice/crio-3c498e481d826d893d71d3b8b90b1a6897acdf35188abed0e5bc72486e9b43f6 WatchSource:0}: Error finding container 3c498e481d826d893d71d3b8b90b1a6897acdf35188abed0e5bc72486e9b43f6: Status 404 returned error can't find the container with id 3c498e481d826d893d71d3b8b90b1a6897acdf35188abed0e5bc72486e9b43f6 Jul 21 04:17:14.728403 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:14.728302 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" event={"ID":"0218007d-5768-4855-ac24-f35d5daf2d66","Type":"ContainerStarted","Data":"3c498e481d826d893d71d3b8b90b1a6897acdf35188abed0e5bc72486e9b43f6"} Jul 21 04:17:17.164250 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:17.164201 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:17:17.164605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:17.164297 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:17:17.164605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:17.164353 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:17:17.747281 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:17.747244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" event={"ID":"0218007d-5768-4855-ac24-f35d5daf2d66","Type":"ContainerStarted","Data":"a52205fb309782413471cbe39792e300082ee073cf6e70f0245887c85a1204cd"} Jul 21 04:17:17.768366 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:17.768305 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" podStartSLOduration=2.077011821 podStartE2EDuration="4.768291995s" totalImagesPullingTime="2.691280174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:17:17.76652244 +0000 UTC m=+483.165855199" watchObservedRunningTime="2026-07-21 04:17:17.768291995 +0000 UTC m=+483.167624752" Jul 21 04:17:18.323352 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:18.323303 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:18.328169 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:18.328144 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:18.753200 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:18.753108 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:18.754164 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:18.754143 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:17:27.786525 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:27.786486 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-c8tn6"] Jul 21 04:17:27.799301 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:27.799271 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:27.799532 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:27.799460 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-c8tn6"] Jul 21 04:17:27.803958 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:27.803933 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 04:17:27.885009 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:27.884978 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-c8tn6"] Jul 21 04:17:27.918792 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:27.918762 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/afb508c8-72c5-4c5a-91e6-79fbe170feb5-config-file\") pod \"limitador-limitador-64c94fd64f-c8tn6\" (UID: \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:27.918969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:27.918870 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd9th\" (UniqueName: \"kubernetes.io/projected/afb508c8-72c5-4c5a-91e6-79fbe170feb5-kube-api-access-nd9th\") pod \"limitador-limitador-64c94fd64f-c8tn6\" (UID: \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:28.019538 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.019502 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9th\" (UniqueName: \"kubernetes.io/projected/afb508c8-72c5-4c5a-91e6-79fbe170feb5-kube-api-access-nd9th\") pod \"limitador-limitador-64c94fd64f-c8tn6\" (UID: \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:28.019698 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.019584 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/afb508c8-72c5-4c5a-91e6-79fbe170feb5-config-file\") pod \"limitador-limitador-64c94fd64f-c8tn6\" (UID: \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:28.020203 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.020183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/afb508c8-72c5-4c5a-91e6-79fbe170feb5-config-file\") pod \"limitador-limitador-64c94fd64f-c8tn6\" (UID: \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:28.030048 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.030015 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9th\" (UniqueName: \"kubernetes.io/projected/afb508c8-72c5-4c5a-91e6-79fbe170feb5-kube-api-access-nd9th\") pod \"limitador-limitador-64c94fd64f-c8tn6\" (UID: \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:28.109952 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.109870 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:28.181830 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.181794 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:17:28.186467 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.186446 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:28.196945 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.196712 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:17:28.222683 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.222653 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:17:28.250499 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.250477 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-c8tn6"] Jul 21 04:17:28.252326 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:17:28.252298 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb508c8_72c5_4c5a_91e6_79fbe170feb5.slice/crio-d6ab5f2c7d9bb7e5eab8653f857c0f54de1c4255227894154c2bda7985c03343 WatchSource:0}: Error finding container d6ab5f2c7d9bb7e5eab8653f857c0f54de1c4255227894154c2bda7985c03343: Status 404 returned error can't find the container with id d6ab5f2c7d9bb7e5eab8653f857c0f54de1c4255227894154c2bda7985c03343 Jul 21 04:17:28.322477 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.322453 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvz8\" (UniqueName: \"kubernetes.io/projected/f93e01da-f748-4c81-9234-812c9acc8772-kube-api-access-vtvz8\") pod \"limitador-limitador-54d6d988d5-vd9b5\" (UID: \"f93e01da-f748-4c81-9234-812c9acc8772\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:28.322583 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.322517 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f93e01da-f748-4c81-9234-812c9acc8772-config-file\") pod \"limitador-limitador-54d6d988d5-vd9b5\" (UID: \"f93e01da-f748-4c81-9234-812c9acc8772\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:28.423123 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.423061 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvz8\" (UniqueName: \"kubernetes.io/projected/f93e01da-f748-4c81-9234-812c9acc8772-kube-api-access-vtvz8\") pod \"limitador-limitador-54d6d988d5-vd9b5\" (UID: \"f93e01da-f748-4c81-9234-812c9acc8772\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:28.423230 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.423137 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f93e01da-f748-4c81-9234-812c9acc8772-config-file\") pod \"limitador-limitador-54d6d988d5-vd9b5\" (UID: \"f93e01da-f748-4c81-9234-812c9acc8772\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:28.423753 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.423733 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f93e01da-f748-4c81-9234-812c9acc8772-config-file\") pod \"limitador-limitador-54d6d988d5-vd9b5\" (UID: \"f93e01da-f748-4c81-9234-812c9acc8772\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:28.431844 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.431818 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvz8\" (UniqueName: \"kubernetes.io/projected/f93e01da-f748-4c81-9234-812c9acc8772-kube-api-access-vtvz8\") pod \"limitador-limitador-54d6d988d5-vd9b5\" (UID: \"f93e01da-f748-4c81-9234-812c9acc8772\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:28.498416 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.498383 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:28.634741 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.634712 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:17:28.636325 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:17:28.636290 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93e01da_f748_4c81_9234_812c9acc8772.slice/crio-6153bc31657c26a6aeb7e3bb2fc937ee10094d2781ce135541a36104bc4d7add WatchSource:0}: Error finding container 6153bc31657c26a6aeb7e3bb2fc937ee10094d2781ce135541a36104bc4d7add: Status 404 returned error can't find the container with id 6153bc31657c26a6aeb7e3bb2fc937ee10094d2781ce135541a36104bc4d7add Jul 21 04:17:28.810073 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.810039 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" event={"ID":"afb508c8-72c5-4c5a-91e6-79fbe170feb5","Type":"ContainerStarted","Data":"d6ab5f2c7d9bb7e5eab8653f857c0f54de1c4255227894154c2bda7985c03343"} Jul 21 04:17:28.811616 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:28.811585 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" event={"ID":"f93e01da-f748-4c81-9234-812c9acc8772","Type":"ContainerStarted","Data":"6153bc31657c26a6aeb7e3bb2fc937ee10094d2781ce135541a36104bc4d7add"} Jul 21 04:17:31.831669 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:31.831626 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" event={"ID":"afb508c8-72c5-4c5a-91e6-79fbe170feb5","Type":"ContainerStarted","Data":"f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14"} Jul 21 04:17:31.832222 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:31.831720 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:31.833328 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:31.833305 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" event={"ID":"f93e01da-f748-4c81-9234-812c9acc8772","Type":"ContainerStarted","Data":"db76c98729bf6db07838054584d5714518237a628091c2937ded4fc9bf8fd94a"} Jul 21 04:17:31.833482 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:31.833401 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:31.850812 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:31.850767 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" podStartSLOduration=1.984454868 podStartE2EDuration="4.850753796s" totalImagesPullingTime="2.866298928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:17:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:17:31.84810178 +0000 UTC m=+497.247434539" watchObservedRunningTime="2026-07-21 04:17:31.850753796 +0000 UTC m=+497.250086554" Jul 21 04:17:31.864880 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:31.864834 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" podStartSLOduration=1.375903938 podStartE2EDuration="3.864820749s" totalImagesPullingTime="2.488916811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:17:31.86281198 +0000 UTC m=+497.262144737" watchObservedRunningTime="2026-07-21 04:17:31.864820749 +0000 UTC m=+497.264153892" Jul 21 04:17:42.839901 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:42.839862 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vd9b5" Jul 21 04:17:42.840381 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:42.840043 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:42.904704 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:42.904666 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-c8tn6"] Jul 21 04:17:42.904870 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:42.904849 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" podUID="afb508c8-72c5-4c5a-91e6-79fbe170feb5" containerName="limitador" containerID="cri-o://f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14" gracePeriod=30 Jul 21 04:17:43.458132 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.458108 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:43.541820 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.541787 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd9th\" (UniqueName: \"kubernetes.io/projected/afb508c8-72c5-4c5a-91e6-79fbe170feb5-kube-api-access-nd9th\") pod \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\" (UID: \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\") " Jul 21 04:17:43.542002 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.541888 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/afb508c8-72c5-4c5a-91e6-79fbe170feb5-config-file\") pod \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\" (UID: \"afb508c8-72c5-4c5a-91e6-79fbe170feb5\") " Jul 21 04:17:43.542259 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.542235 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb508c8-72c5-4c5a-91e6-79fbe170feb5-config-file" (OuterVolumeSpecName: "config-file") pod "afb508c8-72c5-4c5a-91e6-79fbe170feb5" (UID: "afb508c8-72c5-4c5a-91e6-79fbe170feb5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:17:43.543771 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.543745 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb508c8-72c5-4c5a-91e6-79fbe170feb5-kube-api-access-nd9th" (OuterVolumeSpecName: "kube-api-access-nd9th") pod "afb508c8-72c5-4c5a-91e6-79fbe170feb5" (UID: "afb508c8-72c5-4c5a-91e6-79fbe170feb5"). InnerVolumeSpecName "kube-api-access-nd9th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:17:43.642819 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.642786 2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/afb508c8-72c5-4c5a-91e6-79fbe170feb5-config-file\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:17:43.642819 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.642817 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd9th\" (UniqueName: \"kubernetes.io/projected/afb508c8-72c5-4c5a-91e6-79fbe170feb5-kube-api-access-nd9th\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:17:43.898614 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.898527 2567 generic.go:356] "Generic (PLEG): container finished" podID="afb508c8-72c5-4c5a-91e6-79fbe170feb5" containerID="f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14" exitCode=0 Jul 21 04:17:43.898614 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.898569 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" Jul 21 04:17:43.899099 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.898618 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" event={"ID":"afb508c8-72c5-4c5a-91e6-79fbe170feb5","Type":"ContainerDied","Data":"f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14"} Jul 21 04:17:43.899099 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.898673 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-c8tn6" event={"ID":"afb508c8-72c5-4c5a-91e6-79fbe170feb5","Type":"ContainerDied","Data":"d6ab5f2c7d9bb7e5eab8653f857c0f54de1c4255227894154c2bda7985c03343"} Jul 21 04:17:43.899099 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.898708 2567 scope.go:117] "RemoveContainer" containerID="f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14" Jul 21 04:17:43.908558 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.908350 2567 scope.go:117] "RemoveContainer" containerID="f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14" Jul 21 04:17:43.908615 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:17:43.908590 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14\": container with ID starting with f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14 not found: ID does not exist" containerID="f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14" Jul 21 04:17:43.908664 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.908615 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14"} err="failed to get container status \"f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14\": rpc error: code = NotFound desc = could not find container \"f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14\": container with ID starting with f2564ad18aeeac7236e17c2f6c118a78350557ff45fb748f4b95b35c4f6a0e14 not found: ID does not exist" Jul 21 04:17:43.922237 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.922214 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-c8tn6"] Jul 21 04:17:43.928732 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:43.928711 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-c8tn6"] Jul 21 04:17:45.267263 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.267228 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb508c8-72c5-4c5a-91e6-79fbe170feb5" path="/var/lib/kubelet/pods/afb508c8-72c5-4c5a-91e6-79fbe170feb5/volumes" Jul 21 04:17:45.586101 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.586019 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8"] Jul 21 04:17:45.586404 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.586391 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb508c8-72c5-4c5a-91e6-79fbe170feb5" containerName="limitador" Jul 21 04:17:45.586455 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.586405 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb508c8-72c5-4c5a-91e6-79fbe170feb5" containerName="limitador" Jul 21 04:17:45.586491 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.586484 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb508c8-72c5-4c5a-91e6-79fbe170feb5" containerName="limitador" Jul 21 04:17:45.589982 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.589962 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:17:45.592278 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.592248 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 21 04:17:45.592407 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.592350 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 04:17:45.593280 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.593265 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 04:17:45.593327 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.593267 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fl264\"" Jul 21 04:17:45.597446 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.597279 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8"] Jul 21 04:17:45.656647 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.656609 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hk7g\" (UniqueName: \"kubernetes.io/projected/bfa53375-310e-453d-85b0-a01f06ee944c-kube-api-access-7hk7g\") pod \"postgres-69fb4594cf-xrgs8\" (UID: \"bfa53375-310e-453d-85b0-a01f06ee944c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:17:45.656827 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.656660 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bfa53375-310e-453d-85b0-a01f06ee944c-data\") pod \"postgres-69fb4594cf-xrgs8\" (UID: \"bfa53375-310e-453d-85b0-a01f06ee944c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:17:45.757011 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.756975 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hk7g\" (UniqueName: \"kubernetes.io/projected/bfa53375-310e-453d-85b0-a01f06ee944c-kube-api-access-7hk7g\") pod \"postgres-69fb4594cf-xrgs8\" (UID: \"bfa53375-310e-453d-85b0-a01f06ee944c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:17:45.757184 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.757026 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bfa53375-310e-453d-85b0-a01f06ee944c-data\") pod \"postgres-69fb4594cf-xrgs8\" (UID: \"bfa53375-310e-453d-85b0-a01f06ee944c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:17:45.757405 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.757389 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bfa53375-310e-453d-85b0-a01f06ee944c-data\") pod \"postgres-69fb4594cf-xrgs8\" (UID: \"bfa53375-310e-453d-85b0-a01f06ee944c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:17:45.766409 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.766388 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hk7g\" (UniqueName: \"kubernetes.io/projected/bfa53375-310e-453d-85b0-a01f06ee944c-kube-api-access-7hk7g\") pod \"postgres-69fb4594cf-xrgs8\" (UID: \"bfa53375-310e-453d-85b0-a01f06ee944c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:17:45.901014 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:45.900919 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:17:46.062906 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:46.062876 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8"] Jul 21 04:17:46.063041 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:17:46.062980 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa53375_310e_453d_85b0_a01f06ee944c.slice/crio-4783cde584fbea8fc02fee718d104928faba850c689a1de4873b6c1b6fbf6b4a WatchSource:0}: Error finding container 4783cde584fbea8fc02fee718d104928faba850c689a1de4873b6c1b6fbf6b4a: Status 404 returned error can't find the container with id 4783cde584fbea8fc02fee718d104928faba850c689a1de4873b6c1b6fbf6b4a Jul 21 04:17:46.921912 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:46.921832 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" event={"ID":"bfa53375-310e-453d-85b0-a01f06ee944c","Type":"ContainerStarted","Data":"4783cde584fbea8fc02fee718d104928faba850c689a1de4873b6c1b6fbf6b4a"} Jul 21 04:17:51.953963 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:51.953925 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" event={"ID":"bfa53375-310e-453d-85b0-a01f06ee944c","Type":"ContainerStarted","Data":"56897b9141f5843ecc559805f6eba95976ea45626c1366c80fb33cd79fdee9b6"} Jul 21 04:17:51.954363 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:51.953989 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:17:51.973093 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:51.973051 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" podStartSLOduration=2.117221301 podStartE2EDuration="6.973037982s" totalImagesPullingTime="4.855816681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:17:51.971456221 +0000 UTC m=+517.370788976" watchObservedRunningTime="2026-07-21 04:17:51.973037982 +0000 UTC m=+517.372370780" Jul 21 04:17:57.987644 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:17:57.987612 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-xrgs8" Jul 21 04:18:41.397054 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:41.397021 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:18:48.833010 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:48.832971 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rdt7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": read tcp 10.132.0.2:45744->10.132.0.52:15021: read: connection reset by peer" start-of-body= Jul 21 04:18:48.833444 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:48.833060 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" podUID="0218007d-5768-4855-ac24-f35d5daf2d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": read tcp 10.132.0.2:45744->10.132.0.52:15021: read: connection reset by peer" Jul 21 04:18:49.280087 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:49.280048 2567 generic.go:356] "Generic (PLEG): container finished" podID="0218007d-5768-4855-ac24-f35d5daf2d66" containerID="a52205fb309782413471cbe39792e300082ee073cf6e70f0245887c85a1204cd" exitCode=0 Jul 21 04:18:49.280287 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:49.280136 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" event={"ID":"0218007d-5768-4855-ac24-f35d5daf2d66","Type":"ContainerDied","Data":"a52205fb309782413471cbe39792e300082ee073cf6e70f0245887c85a1204cd"} Jul 21 04:18:49.280560 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:49.280546 2567 scope.go:117] "RemoveContainer" containerID="a52205fb309782413471cbe39792e300082ee073cf6e70f0245887c85a1204cd" Jul 21 04:18:49.281571 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:49.281536 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:18:49.281665 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:49.281628 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:18:49.281733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:49.281699 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:18:50.287641 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:50.287604 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" event={"ID":"0218007d-5768-4855-ac24-f35d5daf2d66","Type":"ContainerStarted","Data":"b0fb3c0bf2f38c7edc82840324c8399dcdccd52380b04cfaf406996e88c787ba"} Jul 21 04:18:50.324037 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:50.324003 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:18:50.324208 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:50.324186 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rdt7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 21 04:18:50.324253 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:50.324230 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" podUID="0218007d-5768-4855-ac24-f35d5daf2d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 21 04:18:51.324043 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:51.324004 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rdt7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 21 04:18:51.324563 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:51.324070 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" podUID="0218007d-5768-4855-ac24-f35d5daf2d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 21 04:18:52.323714 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:52.323674 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rdt7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 21 04:18:52.323936 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:52.323737 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" podUID="0218007d-5768-4855-ac24-f35d5daf2d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 21 04:18:53.324428 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:53.324384 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rdt7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 21 04:18:53.324943 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:53.324451 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" podUID="0218007d-5768-4855-ac24-f35d5daf2d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 21 04:18:54.325433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:54.325390 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rdt7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 21 04:18:54.326008 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:54.325973 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" podUID="0218007d-5768-4855-ac24-f35d5daf2d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 21 04:18:55.327673 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:55.327644 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:18:55.328092 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:55.327925 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:18:55.328769 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:18:55.328750 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rdt7k" Jul 21 04:19:04.086979 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.086946 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q"] Jul 21 04:19:04.090139 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.090122 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.092671 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.092644 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 04:19:04.092671 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.092644 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ftzmb\"" Jul 21 04:19:04.092856 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.092684 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 04:19:04.093641 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.093620 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 21 04:19:04.101096 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.101073 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q"] Jul 21 04:19:04.146105 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.146074 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.146105 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.146112 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.146361 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.146143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsnfk\" (UniqueName: \"kubernetes.io/projected/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-kube-api-access-fsnfk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.146361 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.146280 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.146361 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.146312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.146488 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.146408 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.247592 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.247551 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.247592 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.247595 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.247839 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.247613 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.247839 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.247631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsnfk\" (UniqueName: \"kubernetes.io/projected/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-kube-api-access-fsnfk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.247839 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.247679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.247839 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.247699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.248053 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.248016 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.248119 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.248096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.248163 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.248145 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.249996 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.249973 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.250274 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.250256 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.257068 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.257050 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsnfk\" (UniqueName: \"kubernetes.io/projected/ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6-kube-api-access-fsnfk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q\" (UID: \"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.401183 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.401083 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:04.553978 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.552752 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q"] Jul 21 04:19:04.556808 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:19:04.556780 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8b8ed2_a6d2_4a72_99bd_b41fb555f3e6.slice/crio-08fb7761dd21d43f814ea6da5f9eb8e27e1b862b49c0e3160b6dfd15fe25ee7e WatchSource:0}: Error finding container 08fb7761dd21d43f814ea6da5f9eb8e27e1b862b49c0e3160b6dfd15fe25ee7e: Status 404 returned error can't find the container with id 08fb7761dd21d43f814ea6da5f9eb8e27e1b862b49c0e3160b6dfd15fe25ee7e Jul 21 04:19:04.688821 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:04.688727 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:19:05.373503 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:05.373461 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" event={"ID":"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6","Type":"ContainerStarted","Data":"08fb7761dd21d43f814ea6da5f9eb8e27e1b862b49c0e3160b6dfd15fe25ee7e"} Jul 21 04:19:06.380615 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:06.380578 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" event={"ID":"ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6","Type":"ContainerStarted","Data":"801ec4fa816acb7f97eebeea40d3f5374c66a176103fa4af20e29143dd5ee3f1"} Jul 21 04:19:06.400822 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:06.400761 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" podStartSLOduration=1.3954572 podStartE2EDuration="2.40074244s" totalImagesPullingTime="1.00528524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:19:06.398725946 +0000 UTC m=+591.798058716" watchObservedRunningTime="2026-07-21 04:19:06.40074244 +0000 UTC m=+591.800075198" Jul 21 04:19:14.402032 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:14.401992 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:14.402503 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:14.402073 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:14.414675 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:14.414644 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:14.435389 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:14.435357 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q" Jul 21 04:19:15.038065 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:15.038032 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:19:15.044848 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:15.044823 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:19:29.681479 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:29.681443 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:19:45.080602 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.080517 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk"] Jul 21 04:19:45.084451 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.084420 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.086990 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.086965 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 21 04:19:45.093470 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.093444 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk"] Jul 21 04:19:45.111191 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.111153 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.111370 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.111284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr5xc\" (UniqueName: \"kubernetes.io/projected/ffce86ca-f434-4191-ac65-1da3b88b22c7-kube-api-access-dr5xc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.111370 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.111315 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffce86ca-f434-4191-ac65-1da3b88b22c7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.111370 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.111365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.111532 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.111394 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.111532 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.111417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.212225 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.212184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.212225 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.212221 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.212528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.212245 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.212528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.212319 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr5xc\" (UniqueName: \"kubernetes.io/projected/ffce86ca-f434-4191-ac65-1da3b88b22c7-kube-api-access-dr5xc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.212528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.212366 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffce86ca-f434-4191-ac65-1da3b88b22c7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.212528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.212409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.212757 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.212704 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.212801 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.212762 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.212801 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.212790 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.214512 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.214486 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffce86ca-f434-4191-ac65-1da3b88b22c7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.214861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.214842 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffce86ca-f434-4191-ac65-1da3b88b22c7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.220950 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.220917 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr5xc\" (UniqueName: \"kubernetes.io/projected/ffce86ca-f434-4191-ac65-1da3b88b22c7-kube-api-access-dr5xc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s6nsk\" (UID: \"ffce86ca-f434-4191-ac65-1da3b88b22c7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.396806 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.396713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:45.535475 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.535448 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk"] Jul 21 04:19:45.536893 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:19:45.536866 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffce86ca_f434_4191_ac65_1da3b88b22c7.slice/crio-b4f02f186665369b0ed5b23a59e1caa850a44bd5bcb9cd323aadbf00f3f158f7 WatchSource:0}: Error finding container b4f02f186665369b0ed5b23a59e1caa850a44bd5bcb9cd323aadbf00f3f158f7: Status 404 returned error can't find the container with id b4f02f186665369b0ed5b23a59e1caa850a44bd5bcb9cd323aadbf00f3f158f7 Jul 21 04:19:45.538353 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.538323 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 04:19:45.594766 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:45.594708 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" event={"ID":"ffce86ca-f434-4191-ac65-1da3b88b22c7","Type":"ContainerStarted","Data":"b4f02f186665369b0ed5b23a59e1caa850a44bd5bcb9cd323aadbf00f3f158f7"} Jul 21 04:19:46.272110 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:46.272074 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:19:46.601808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:46.601716 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" event={"ID":"ffce86ca-f434-4191-ac65-1da3b88b22c7","Type":"ContainerStarted","Data":"4447166e0c066beb3854727f437163e667fa532ffddd910b38901d68863d5e4f"} Jul 21 04:19:46.620101 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:46.620035 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" podStartSLOduration=1.449049783 podStartE2EDuration="1.620013665s" totalImagesPullingTime="170.963882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:19:46.619475186 +0000 UTC m=+632.018807953" watchObservedRunningTime="2026-07-21 04:19:46.620013665 +0000 UTC m=+632.019346424" Jul 21 04:19:55.397086 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:55.397049 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:55.397538 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:55.397109 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:55.409215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:55.409190 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:55.663925 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:55.663840 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s6nsk" Jul 21 04:19:56.578282 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:19:56.578242 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:20:13.172561 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:20:13.172514 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:20:24.999037 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:20:24.998994 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:21:45.978146 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:21:45.978110 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:21:57.577428 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:21:57.577389 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:22:07.682824 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:22:07.682790 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:23:46.485325 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:23:46.485291 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:23:57.275095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:23:57.275060 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:24:15.075177 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:24:15.075147 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:24:15.078856 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:24:15.078828 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:24:34.083609 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:24:34.083576 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:24:43.679312 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:24:43.679274 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:24:53.983060 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:24:53.983018 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:25:02.373518 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:25:02.373478 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:25:11.679867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:25:11.679831 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:25:22.802444 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:25:22.802405 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:25:33.400526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:25:33.396221 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:25:43.286112 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:25:43.286026 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:26:04.575845 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:26:04.575807 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:26:16.878902 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:26:16.878867 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:26:39.479671 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:26:39.479634 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:26:51.077858 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:26:51.077823 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:27:05.989272 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:27:05.989232 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:27:15.394901 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:27:15.394870 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:27:27.290589 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:27:27.290548 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:27:36.683835 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:27:36.683797 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:27:46.378383 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:27:46.378327 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:28:10.666688 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:10.666650 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp"] Jul 21 04:28:10.667292 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:10.666949 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" podUID="179c9a09-3ff7-431f-9305-5fb4bbe5f128" containerName="manager" containerID="cri-o://1a7262233f429f31db669aff9b12beb5d09f30f7febc79fc63f0684fddcd4069" gracePeriod=10 Jul 21 04:28:13.402684 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:13.402651 2567 generic.go:356] "Generic (PLEG): container finished" podID="179c9a09-3ff7-431f-9305-5fb4bbe5f128" containerID="1a7262233f429f31db669aff9b12beb5d09f30f7febc79fc63f0684fddcd4069" exitCode=0 Jul 21 04:28:13.403077 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:13.402722 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" event={"ID":"179c9a09-3ff7-431f-9305-5fb4bbe5f128","Type":"ContainerDied","Data":"1a7262233f429f31db669aff9b12beb5d09f30f7febc79fc63f0684fddcd4069"} Jul 21 04:28:13.823686 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:13.823662 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:28:13.906263 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:13.906232 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/179c9a09-3ff7-431f-9305-5fb4bbe5f128-extensions-socket-volume\") pod \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\" (UID: \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\") " Jul 21 04:28:13.906463 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:13.906326 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-222s2\" (UniqueName: \"kubernetes.io/projected/179c9a09-3ff7-431f-9305-5fb4bbe5f128-kube-api-access-222s2\") pod \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\" (UID: \"179c9a09-3ff7-431f-9305-5fb4bbe5f128\") " Jul 21 04:28:13.906611 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:13.906589 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179c9a09-3ff7-431f-9305-5fb4bbe5f128-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "179c9a09-3ff7-431f-9305-5fb4bbe5f128" (UID: "179c9a09-3ff7-431f-9305-5fb4bbe5f128"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:28:13.908275 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:13.908253 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179c9a09-3ff7-431f-9305-5fb4bbe5f128-kube-api-access-222s2" (OuterVolumeSpecName: "kube-api-access-222s2") pod "179c9a09-3ff7-431f-9305-5fb4bbe5f128" (UID: "179c9a09-3ff7-431f-9305-5fb4bbe5f128"). InnerVolumeSpecName "kube-api-access-222s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:28:14.007280 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:14.007207 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-222s2\" (UniqueName: \"kubernetes.io/projected/179c9a09-3ff7-431f-9305-5fb4bbe5f128-kube-api-access-222s2\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:28:14.007280 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:14.007234 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/179c9a09-3ff7-431f-9305-5fb4bbe5f128-extensions-socket-volume\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:28:14.408933 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:14.408909 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" Jul 21 04:28:14.409352 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:14.408955 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp" event={"ID":"179c9a09-3ff7-431f-9305-5fb4bbe5f128","Type":"ContainerDied","Data":"920d150f69d285177a2af82c88203713e8982cd0d628e6113813e6c9b9c9db9b"} Jul 21 04:28:14.409352 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:14.409022 2567 scope.go:117] "RemoveContainer" containerID="1a7262233f429f31db669aff9b12beb5d09f30f7febc79fc63f0684fddcd4069" Jul 21 04:28:14.438699 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:14.438670 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp"] Jul 21 04:28:14.440910 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:14.440890 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5d7jp"] Jul 21 04:28:15.265953 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:28:15.265925 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="179c9a09-3ff7-431f-9305-5fb4bbe5f128" path="/var/lib/kubelet/pods/179c9a09-3ff7-431f-9305-5fb4bbe5f128/volumes" Jul 21 04:29:11.862233 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:11.862200 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd"] Jul 21 04:29:11.864553 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:11.862612 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179c9a09-3ff7-431f-9305-5fb4bbe5f128" containerName="manager" Jul 21 04:29:11.864553 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:11.862626 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="179c9a09-3ff7-431f-9305-5fb4bbe5f128" containerName="manager" Jul 21 04:29:11.864553 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:11.862689 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="179c9a09-3ff7-431f-9305-5fb4bbe5f128" containerName="manager" Jul 21 04:29:11.865531 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:11.865514 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:11.868082 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:11.868064 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jnblz\"" Jul 21 04:29:11.879288 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:11.879264 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd"] Jul 21 04:29:11.922652 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:11.922617 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f418020-20c1-496e-b74e-03845819655d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2zxbd\" (UID: \"3f418020-20c1-496e-b74e-03845819655d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:11.922652 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:11.922655 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8b47\" (UniqueName: \"kubernetes.io/projected/3f418020-20c1-496e-b74e-03845819655d-kube-api-access-g8b47\") pod \"kuadrant-operator-controller-manager-86878c769f-2zxbd\" (UID: \"3f418020-20c1-496e-b74e-03845819655d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:12.023946 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.023916 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f418020-20c1-496e-b74e-03845819655d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2zxbd\" (UID: \"3f418020-20c1-496e-b74e-03845819655d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:12.024092 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.023954 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8b47\" (UniqueName: \"kubernetes.io/projected/3f418020-20c1-496e-b74e-03845819655d-kube-api-access-g8b47\") pod \"kuadrant-operator-controller-manager-86878c769f-2zxbd\" (UID: \"3f418020-20c1-496e-b74e-03845819655d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:12.024408 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.024388 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f418020-20c1-496e-b74e-03845819655d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2zxbd\" (UID: \"3f418020-20c1-496e-b74e-03845819655d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:12.033492 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.033456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8b47\" (UniqueName: \"kubernetes.io/projected/3f418020-20c1-496e-b74e-03845819655d-kube-api-access-g8b47\") pod \"kuadrant-operator-controller-manager-86878c769f-2zxbd\" (UID: \"3f418020-20c1-496e-b74e-03845819655d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:12.175600 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.175509 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:12.322719 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.322688 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd"] Jul 21 04:29:12.322846 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:29:12.322819 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f418020_20c1_496e_b74e_03845819655d.slice/crio-40ae48bcaa948a9cb84158ce4ba8bcf8abf21b1b7541f96b98b2f5cff5d9fd41 WatchSource:0}: Error finding container 40ae48bcaa948a9cb84158ce4ba8bcf8abf21b1b7541f96b98b2f5cff5d9fd41: Status 404 returned error can't find the container with id 40ae48bcaa948a9cb84158ce4ba8bcf8abf21b1b7541f96b98b2f5cff5d9fd41 Jul 21 04:29:12.724686 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.724646 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" event={"ID":"3f418020-20c1-496e-b74e-03845819655d","Type":"ContainerStarted","Data":"8e43868f0a0707bc419900026366fea3f92d2358bbc6b790ca87395823740676"} Jul 21 04:29:12.724901 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.724711 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" event={"ID":"3f418020-20c1-496e-b74e-03845819655d","Type":"ContainerStarted","Data":"40ae48bcaa948a9cb84158ce4ba8bcf8abf21b1b7541f96b98b2f5cff5d9fd41"} Jul 21 04:29:12.724901 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.724769 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:12.746846 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:12.746803 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" podStartSLOduration=1.746786234 podStartE2EDuration="1.746786234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:29:12.746260172 +0000 UTC m=+1198.145592930" watchObservedRunningTime="2026-07-21 04:29:12.746786234 +0000 UTC m=+1198.146118993" Jul 21 04:29:15.114876 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:15.114847 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:29:15.115389 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:15.115028 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:29:23.732349 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:23.732302 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2zxbd" Jul 21 04:29:32.581097 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:32.581058 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:29:41.476476 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:41.476436 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:29:54.281874 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:29:54.281840 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:30:05.878992 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:30:05.878953 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:30:16.875123 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:30:16.875088 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:30:27.479505 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:30:27.479467 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:30:38.084366 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:30:38.084316 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:30:47.682710 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:30:47.682673 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:30:59.579870 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:30:59.579833 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:31:08.378830 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:08.378796 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:31:18.982572 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:18.982537 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:31:40.585578 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:40.585492 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:31:49.588733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:49.588685 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:31:59.384519 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.384482 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct"] Jul 21 04:31:59.388187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.388163 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.390795 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.390770 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fa6b59de-kserve-self-signed-certs\"" Jul 21 04:31:59.406427 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.406403 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct"] Jul 21 04:31:59.489368 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.489299 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-home\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.489368 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.489364 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-dshm\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.489623 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.489462 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nth9h\" (UniqueName: \"kubernetes.io/projected/164a12bb-302f-4acb-8b1d-28076a8f15a4-kube-api-access-nth9h\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.489623 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.489512 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-model-cache\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.489623 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.489548 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-tmp-dir\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.489623 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.489577 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/164a12bb-302f-4acb-8b1d-28076a8f15a4-tls-certs\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.590672 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.590640 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-home\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.590866 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.590685 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-dshm\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.590866 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.590734 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nth9h\" (UniqueName: \"kubernetes.io/projected/164a12bb-302f-4acb-8b1d-28076a8f15a4-kube-api-access-nth9h\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.590866 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.590762 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-model-cache\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.590866 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.590793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-tmp-dir\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.590866 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.590852 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/164a12bb-302f-4acb-8b1d-28076a8f15a4-tls-certs\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.591142 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.591043 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-home\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.591142 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.591133 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-tmp-dir\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.591253 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.591175 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-model-cache\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.593225 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.593201 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-dshm\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.593512 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.593492 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/164a12bb-302f-4acb-8b1d-28076a8f15a4-tls-certs\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.599859 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.599837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nth9h\" (UniqueName: \"kubernetes.io/projected/164a12bb-302f-4acb-8b1d-28076a8f15a4-kube-api-access-nth9h\") pod \"e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.697497 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.697409 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:31:59.836735 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:31:59.836708 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct"] Jul 21 04:31:59.838598 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:31:59.838566 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod164a12bb_302f_4acb_8b1d_28076a8f15a4.slice/crio-96ebdeed48d7ab1b453e75ef2d895a85170adc9aa8f493a9d196e200beb97928 WatchSource:0}: Error finding container 96ebdeed48d7ab1b453e75ef2d895a85170adc9aa8f493a9d196e200beb97928: Status 404 returned error can't find the container with id 96ebdeed48d7ab1b453e75ef2d895a85170adc9aa8f493a9d196e200beb97928 Jul 21 04:32:00.661153 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:00.661112 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" event={"ID":"164a12bb-302f-4acb-8b1d-28076a8f15a4","Type":"ContainerStarted","Data":"bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727"} Jul 21 04:32:00.661153 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:00.661164 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" event={"ID":"164a12bb-302f-4acb-8b1d-28076a8f15a4","Type":"ContainerStarted","Data":"96ebdeed48d7ab1b453e75ef2d895a85170adc9aa8f493a9d196e200beb97928"} Jul 21 04:32:00.683483 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:00.683432 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" podStartSLOduration=1.683417279 podStartE2EDuration="1.683417279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:32:00.681402358 +0000 UTC m=+1366.080735116" watchObservedRunningTime="2026-07-21 04:32:00.683417279 +0000 UTC m=+1366.082750036" Jul 21 04:32:02.879371 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:02.879324 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:32:09.697986 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:09.697938 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:32:09.698580 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:09.698064 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:32:09.710685 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:09.710662 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:32:09.726274 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:09.726250 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:32:15.276357 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.276314 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv"] Jul 21 04:32:15.279967 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.279944 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.282410 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.282388 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fa6b59de-kserve-self-signed-certs\"" Jul 21 04:32:15.290292 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.290256 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv"] Jul 21 04:32:15.333321 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.333286 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-model-cache\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.333504 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.333326 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6106299-f433-4db2-8c52-03ff5a051038-tls-certs\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.333504 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.333394 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-home\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.333504 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.333412 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-tmp-dir\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.333504 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.333431 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-dshm\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.333504 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.333447 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsl82\" (UniqueName: \"kubernetes.io/projected/f6106299-f433-4db2-8c52-03ff5a051038-kube-api-access-zsl82\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.434749 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.434712 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-home\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.434930 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.434763 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-tmp-dir\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.434930 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.434799 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-dshm\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.434930 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.434825 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsl82\" (UniqueName: \"kubernetes.io/projected/f6106299-f433-4db2-8c52-03ff5a051038-kube-api-access-zsl82\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.434930 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.434920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-model-cache\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.435149 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.434950 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6106299-f433-4db2-8c52-03ff5a051038-tls-certs\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.435215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.435144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-home\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.435268 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.435226 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-tmp-dir\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.435433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.435414 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-model-cache\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.437195 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.437173 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-dshm\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.437397 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.437382 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6106299-f433-4db2-8c52-03ff5a051038-tls-certs\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.443522 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.443498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsl82\" (UniqueName: \"kubernetes.io/projected/f6106299-f433-4db2-8c52-03ff5a051038-kube-api-access-zsl82\") pod \"e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.590940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.590848 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:15.736180 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.736149 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv"] Jul 21 04:32:15.738047 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:32:15.738014 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6106299_f433_4db2_8c52_03ff5a051038.slice/crio-96413b040f177b446be8949f88872b6b58518ce69b4eef8cb2b88906345389b6 WatchSource:0}: Error finding container 96413b040f177b446be8949f88872b6b58518ce69b4eef8cb2b88906345389b6: Status 404 returned error can't find the container with id 96413b040f177b446be8949f88872b6b58518ce69b4eef8cb2b88906345389b6 Jul 21 04:32:15.752854 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:15.752828 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" event={"ID":"f6106299-f433-4db2-8c52-03ff5a051038","Type":"ContainerStarted","Data":"96413b040f177b446be8949f88872b6b58518ce69b4eef8cb2b88906345389b6"} Jul 21 04:32:16.760351 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:16.760301 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" event={"ID":"f6106299-f433-4db2-8c52-03ff5a051038","Type":"ContainerStarted","Data":"88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58"} Jul 21 04:32:16.779541 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:16.779486 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" podStartSLOduration=1.779467779 podStartE2EDuration="1.779467779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:32:16.777298631 +0000 UTC m=+1382.176631396" watchObservedRunningTime="2026-07-21 04:32:16.779467779 +0000 UTC m=+1382.178800537" Jul 21 04:32:18.781041 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:18.781003 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:32:25.591403 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:25.591366 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:25.591403 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:25.591414 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:25.605934 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:25.605909 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:25.819785 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:25.819747 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:37.573034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:37.573001 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct"] Jul 21 04:32:38.513375 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:38.513325 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv"] Jul 21 04:32:41.486359 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:41.483447 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:32:52.593382 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.593322 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" podUID="164a12bb-302f-4acb-8b1d-28076a8f15a4" containerName="main" containerID="cri-o://bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727" gracePeriod=45 Jul 21 04:32:52.849564 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.849504 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:32:52.878671 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.878638 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-home\") pod \"164a12bb-302f-4acb-8b1d-28076a8f15a4\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " Jul 21 04:32:52.878817 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.878702 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nth9h\" (UniqueName: \"kubernetes.io/projected/164a12bb-302f-4acb-8b1d-28076a8f15a4-kube-api-access-nth9h\") pod \"164a12bb-302f-4acb-8b1d-28076a8f15a4\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " Jul 21 04:32:52.878817 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.878734 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-model-cache\") pod \"164a12bb-302f-4acb-8b1d-28076a8f15a4\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " Jul 21 04:32:52.878817 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.878756 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/164a12bb-302f-4acb-8b1d-28076a8f15a4-tls-certs\") pod \"164a12bb-302f-4acb-8b1d-28076a8f15a4\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " Jul 21 04:32:52.878817 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.878788 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-tmp-dir\") pod \"164a12bb-302f-4acb-8b1d-28076a8f15a4\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " Jul 21 04:32:52.879033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.878821 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-dshm\") pod \"164a12bb-302f-4acb-8b1d-28076a8f15a4\" (UID: \"164a12bb-302f-4acb-8b1d-28076a8f15a4\") " Jul 21 04:32:52.879033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.878926 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-home" (OuterVolumeSpecName: "home") pod "164a12bb-302f-4acb-8b1d-28076a8f15a4" (UID: "164a12bb-302f-4acb-8b1d-28076a8f15a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:32:52.879140 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.879026 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "164a12bb-302f-4acb-8b1d-28076a8f15a4" (UID: "164a12bb-302f-4acb-8b1d-28076a8f15a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:32:52.879140 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.879053 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "164a12bb-302f-4acb-8b1d-28076a8f15a4" (UID: "164a12bb-302f-4acb-8b1d-28076a8f15a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:32:52.879240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.879199 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-home\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:52.879240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.879219 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-model-cache\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:52.879240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.879235 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-tmp-dir\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:52.881129 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.881107 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-dshm" (OuterVolumeSpecName: "dshm") pod "164a12bb-302f-4acb-8b1d-28076a8f15a4" (UID: "164a12bb-302f-4acb-8b1d-28076a8f15a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:32:52.881222 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.881175 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164a12bb-302f-4acb-8b1d-28076a8f15a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "164a12bb-302f-4acb-8b1d-28076a8f15a4" (UID: "164a12bb-302f-4acb-8b1d-28076a8f15a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:32:52.881300 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.881283 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164a12bb-302f-4acb-8b1d-28076a8f15a4-kube-api-access-nth9h" (OuterVolumeSpecName: "kube-api-access-nth9h") pod "164a12bb-302f-4acb-8b1d-28076a8f15a4" (UID: "164a12bb-302f-4acb-8b1d-28076a8f15a4"). InnerVolumeSpecName "kube-api-access-nth9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:32:52.967949 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.967917 2567 generic.go:356] "Generic (PLEG): container finished" podID="164a12bb-302f-4acb-8b1d-28076a8f15a4" containerID="bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727" exitCode=0 Jul 21 04:32:52.968130 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.967966 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" Jul 21 04:32:52.968130 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.967994 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" event={"ID":"164a12bb-302f-4acb-8b1d-28076a8f15a4","Type":"ContainerDied","Data":"bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727"} Jul 21 04:32:52.968130 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.968036 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct" event={"ID":"164a12bb-302f-4acb-8b1d-28076a8f15a4","Type":"ContainerDied","Data":"96ebdeed48d7ab1b453e75ef2d895a85170adc9aa8f493a9d196e200beb97928"} Jul 21 04:32:52.968130 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.968056 2567 scope.go:117] "RemoveContainer" containerID="bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727" Jul 21 04:32:52.977187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.977170 2567 scope.go:117] "RemoveContainer" containerID="bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727" Jul 21 04:32:52.977462 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:32:52.977442 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727\": container with ID starting with bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727 not found: ID does not exist" containerID="bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727" Jul 21 04:32:52.977536 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.977470 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727"} err="failed to get container status \"bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727\": rpc error: code = NotFound desc = could not find container \"bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727\": container with ID starting with bca1ca52636b98951656724832619df37c7bf8348bf5747e389751d4b09a8727 not found: ID does not exist" Jul 21 04:32:52.979962 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.979936 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nth9h\" (UniqueName: \"kubernetes.io/projected/164a12bb-302f-4acb-8b1d-28076a8f15a4-kube-api-access-nth9h\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:52.980066 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.979964 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/164a12bb-302f-4acb-8b1d-28076a8f15a4-tls-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:52.980066 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.979979 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/164a12bb-302f-4acb-8b1d-28076a8f15a4-dshm\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:52.992470 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.992446 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct"] Jul 21 04:32:52.996934 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:52.996913 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fa6b59de-kserve-76787c4579-sphct"] Jul 21 04:32:53.267266 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.267238 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="164a12bb-302f-4acb-8b1d-28076a8f15a4" path="/var/lib/kubelet/pods/164a12bb-302f-4acb-8b1d-28076a8f15a4/volumes" Jul 21 04:32:53.534745 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.534660 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" podUID="f6106299-f433-4db2-8c52-03ff5a051038" containerName="main" containerID="cri-o://88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58" gracePeriod=45 Jul 21 04:32:53.795096 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.795028 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:53.888436 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.888404 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-home\") pod \"f6106299-f433-4db2-8c52-03ff5a051038\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " Jul 21 04:32:53.888629 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.888484 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-dshm\") pod \"f6106299-f433-4db2-8c52-03ff5a051038\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " Jul 21 04:32:53.888629 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.888519 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsl82\" (UniqueName: \"kubernetes.io/projected/f6106299-f433-4db2-8c52-03ff5a051038-kube-api-access-zsl82\") pod \"f6106299-f433-4db2-8c52-03ff5a051038\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " Jul 21 04:32:53.888629 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.888574 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-tmp-dir\") pod \"f6106299-f433-4db2-8c52-03ff5a051038\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " Jul 21 04:32:53.888805 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.888643 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-model-cache\") pod \"f6106299-f433-4db2-8c52-03ff5a051038\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " Jul 21 04:32:53.888805 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.888688 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6106299-f433-4db2-8c52-03ff5a051038-tls-certs\") pod \"f6106299-f433-4db2-8c52-03ff5a051038\" (UID: \"f6106299-f433-4db2-8c52-03ff5a051038\") " Jul 21 04:32:53.888805 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.888691 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-home" (OuterVolumeSpecName: "home") pod "f6106299-f433-4db2-8c52-03ff5a051038" (UID: "f6106299-f433-4db2-8c52-03ff5a051038"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:32:53.888940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.888880 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6106299-f433-4db2-8c52-03ff5a051038" (UID: "f6106299-f433-4db2-8c52-03ff5a051038"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:32:53.888940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.888892 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6106299-f433-4db2-8c52-03ff5a051038" (UID: "f6106299-f433-4db2-8c52-03ff5a051038"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:32:53.897033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.889695 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-home\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:53.897033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.889729 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-tmp-dir\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:53.897033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.889748 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-model-cache\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:53.897650 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.897420 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6106299-f433-4db2-8c52-03ff5a051038-kube-api-access-zsl82" (OuterVolumeSpecName: "kube-api-access-zsl82") pod "f6106299-f433-4db2-8c52-03ff5a051038" (UID: "f6106299-f433-4db2-8c52-03ff5a051038"). InnerVolumeSpecName "kube-api-access-zsl82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:32:53.897934 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.897919 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6106299-f433-4db2-8c52-03ff5a051038-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6106299-f433-4db2-8c52-03ff5a051038" (UID: "f6106299-f433-4db2-8c52-03ff5a051038"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:32:53.898004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.897934 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-dshm" (OuterVolumeSpecName: "dshm") pod "f6106299-f433-4db2-8c52-03ff5a051038" (UID: "f6106299-f433-4db2-8c52-03ff5a051038"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:32:53.975318 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.975281 2567 generic.go:356] "Generic (PLEG): container finished" podID="f6106299-f433-4db2-8c52-03ff5a051038" containerID="88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58" exitCode=0 Jul 21 04:32:53.975490 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.975358 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" Jul 21 04:32:53.975490 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.975364 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" event={"ID":"f6106299-f433-4db2-8c52-03ff5a051038","Type":"ContainerDied","Data":"88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58"} Jul 21 04:32:53.975490 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.975406 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv" event={"ID":"f6106299-f433-4db2-8c52-03ff5a051038","Type":"ContainerDied","Data":"96413b040f177b446be8949f88872b6b58518ce69b4eef8cb2b88906345389b6"} Jul 21 04:32:53.975490 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.975424 2567 scope.go:117] "RemoveContainer" containerID="88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58" Jul 21 04:32:53.985125 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.985105 2567 scope.go:117] "RemoveContainer" containerID="88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58" Jul 21 04:32:53.985385 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:32:53.985363 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58\": container with ID starting with 88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58 not found: ID does not exist" containerID="88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58" Jul 21 04:32:53.985471 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.985392 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58"} err="failed to get container status \"88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58\": rpc error: code = NotFound desc = could not find container \"88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58\": container with ID starting with 88ae553f596b3dbed0e7467c9085343d85208ea993058eee0a3b618d3e35dd58 not found: ID does not exist" Jul 21 04:32:53.990867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.990848 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6106299-f433-4db2-8c52-03ff5a051038-tls-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:53.990867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.990867 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6106299-f433-4db2-8c52-03ff5a051038-dshm\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:53.990988 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:53.990876 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsl82\" (UniqueName: \"kubernetes.io/projected/f6106299-f433-4db2-8c52-03ff5a051038-kube-api-access-zsl82\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:32:54.000521 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:54.000499 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv"] Jul 21 04:32:54.004696 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:54.004671 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fa6b59de-kserve-68895bb7c5-crxjv"] Jul 21 04:32:54.777442 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:54.777409 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:32:55.266592 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:32:55.266563 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6106299-f433-4db2-8c52-03ff5a051038" path="/var/lib/kubelet/pods/f6106299-f433-4db2-8c52-03ff5a051038/volumes" Jul 21 04:33:03.488546 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:33:03.488505 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:33:15.687509 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:33:15.687471 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:33:25.180229 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:33:25.180194 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:33:34.479809 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:33:34.479778 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:33:45.184421 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:33:45.184390 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:33:54.498365 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:33:54.498318 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:34:04.078876 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:34:04.078837 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:34:15.150957 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:34:15.150926 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:34:15.153236 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:34:15.152134 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:34:17.583677 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:34:17.583637 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:34:26.485679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:34:26.485644 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:34:36.085957 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:34:36.085919 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:34:45.584501 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:34:45.584415 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:34:58.280115 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:34:58.280080 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:35:07.288152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:35:07.288123 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:35:19.180772 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:35:19.180740 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:35:31.071708 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:35:31.071669 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:35:41.480774 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:35:41.480734 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:35:53.979157 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:35:53.979107 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:36:04.882453 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:04.882416 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:36:14.379434 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:14.379401 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:36:16.668199 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.668170 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt"] Jul 21 04:36:16.668586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.668577 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="164a12bb-302f-4acb-8b1d-28076a8f15a4" containerName="main" Jul 21 04:36:16.668645 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.668588 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="164a12bb-302f-4acb-8b1d-28076a8f15a4" containerName="main" Jul 21 04:36:16.668645 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.668597 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6106299-f433-4db2-8c52-03ff5a051038" containerName="main" Jul 21 04:36:16.668645 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.668602 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6106299-f433-4db2-8c52-03ff5a051038" containerName="main" Jul 21 04:36:16.668742 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.668693 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="164a12bb-302f-4acb-8b1d-28076a8f15a4" containerName="main" Jul 21 04:36:16.668742 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.668702 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6106299-f433-4db2-8c52-03ff5a051038" containerName="main" Jul 21 04:36:16.672246 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.672228 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.674665 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.674645 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fc7e6220-openshift-default-dockercfg-nqwcx\"" Jul 21 04:36:16.692385 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.692364 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt"] Jul 21 04:36:16.857029 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.856995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istiod-ca-cert\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.857214 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.857041 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-credential-socket\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.857214 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.857109 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-envoy\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.857214 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.857140 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-token\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.857214 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.857172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-certs\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.857214 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.857202 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-podinfo\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.857467 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.857227 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqxg\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-kube-api-access-hrqxg\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.857467 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.857272 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-socket\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.857467 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.857313 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-data\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.958587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958508 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-socket\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.958587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958552 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-data\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.958587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istiod-ca-cert\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.958834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958606 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-credential-socket\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.958834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-envoy\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.958834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958655 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-token\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.958834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958764 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-certs\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.958834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-podinfo\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.958834 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958834 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqxg\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-kube-api-access-hrqxg\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.959152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958943 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-socket\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.959152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.958965 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-data\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.959527 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.959499 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istiod-ca-cert\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.959674 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.959570 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-credential-socket\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.959674 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.959586 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-certs\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.960908 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.960888 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-envoy\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.961294 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.961278 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-podinfo\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.967309 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.967283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqxg\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-kube-api-access-hrqxg\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.967807 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.967780 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-token\") pod \"e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:16.982764 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:16.982745 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:17.128487 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:17.128452 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt"] Jul 21 04:36:17.129026 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:36:17.129002 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5c6e12_cc79_4d84_a652_c3c12fe03027.slice/crio-a8e09d5bc7446fb48e5f14484c19239b234ce7022e9c032af1846a74f21363ad WatchSource:0}: Error finding container a8e09d5bc7446fb48e5f14484c19239b234ce7022e9c032af1846a74f21363ad: Status 404 returned error can't find the container with id a8e09d5bc7446fb48e5f14484c19239b234ce7022e9c032af1846a74f21363ad Jul 21 04:36:17.131294 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:17.131251 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:36:17.131400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:17.131371 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:36:17.131448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:17.131425 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:36:18.112672 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:18.112626 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" event={"ID":"3a5c6e12-cc79-4d84-a652-c3c12fe03027","Type":"ContainerStarted","Data":"6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448"} Jul 21 04:36:18.113055 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:18.112692 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" event={"ID":"3a5c6e12-cc79-4d84-a652-c3c12fe03027","Type":"ContainerStarted","Data":"a8e09d5bc7446fb48e5f14484c19239b234ce7022e9c032af1846a74f21363ad"} Jul 21 04:36:18.133326 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:18.133278 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" podStartSLOduration=2.133264727 podStartE2EDuration="2.133264727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:36:18.132259159 +0000 UTC m=+1623.531591917" watchObservedRunningTime="2026-07-21 04:36:18.133264727 +0000 UTC m=+1623.532597485" Jul 21 04:36:18.983595 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:18.983560 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:18.988295 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:18.988264 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:19.118514 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:19.118474 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:19.119687 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:19.119664 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:22.475742 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:22.475706 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:36:31.191959 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:31.191927 2567 generic.go:356] "Generic (PLEG): container finished" podID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerID="6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448" exitCode=0 Jul 21 04:36:31.192424 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:31.191999 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" event={"ID":"3a5c6e12-cc79-4d84-a652-c3c12fe03027","Type":"ContainerDied","Data":"6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448"} Jul 21 04:36:31.192424 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:31.192405 2567 scope.go:117] "RemoveContainer" containerID="6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448" Jul 21 04:36:31.193415 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:31.193386 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:36:31.193485 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:31.193457 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:36:31.193522 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:31.193490 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:36:32.202107 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:32.202068 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" event={"ID":"3a5c6e12-cc79-4d84-a652-c3c12fe03027","Type":"ContainerStarted","Data":"f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c"} Jul 21 04:36:32.983948 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:32.983909 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:32.985050 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:32.985022 2567 patch_prober.go:28] interesting pod/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 21 04:36:32.985194 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:32.985077 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 21 04:36:33.983581 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:33.983545 2567 patch_prober.go:28] interesting pod/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 21 04:36:33.983990 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:33.983614 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 21 04:36:34.119140 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:34.119092 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:34.983770 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:34.983731 2567 patch_prober.go:28] interesting pod/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 21 04:36:34.984189 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:34.983786 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 21 04:36:35.987891 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:35.987859 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:35.988826 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:35.988805 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:36:58.589303 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:36:58.589265 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:37:00.375908 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:00.375872 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt"] Jul 21 04:37:00.376380 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:00.376147 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" containerID="cri-o://f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c" gracePeriod=30 Jul 21 04:37:01.408316 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.408283 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn"] Jul 21 04:37:01.411996 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.411977 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.414439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.414415 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c0006c3d-gw-openshift-default-dockercfg-rnjkn\"" Jul 21 04:37:01.424964 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.424941 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn"] Jul 21 04:37:01.483635 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.483598 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-token\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.483808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.483662 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istiod-ca-cert\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.483808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.483694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-envoy\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.483808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.483736 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-credential-socket\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.483808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.483759 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2mx5\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-kube-api-access-n2mx5\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.483808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.483778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-podinfo\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.483982 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.483902 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-socket\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.483982 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.483934 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-certs\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.483982 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.483970 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-data\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.584997 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.584961 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-podinfo\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585182 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585012 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-socket\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585182 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585030 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-certs\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585182 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-data\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585182 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-token\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585182 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585151 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istiod-ca-cert\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585485 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-envoy\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585485 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585391 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-credential-socket\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585485 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585433 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2mx5\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-kube-api-access-n2mx5\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585485 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-certs\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585661 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585506 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-socket\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585661 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585550 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-data\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.585783 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.585766 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-credential-socket\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.586133 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.586110 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istiod-ca-cert\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.587537 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.587519 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-envoy\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.587745 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.587728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-podinfo\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.594275 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.594244 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2mx5\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-kube-api-access-n2mx5\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.594650 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.594630 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-token\") pod \"e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.722564 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.722466 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:01.867820 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.867794 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn"] Jul 21 04:37:01.869499 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:37:01.869470 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09e9457_109b_44f2_8a49_3dc5d72c45ea.slice/crio-997f78fa735b62216b42b2f0f6ec3f1c6c00e2fca949996c7e154a760f6eedc8 WatchSource:0}: Error finding container 997f78fa735b62216b42b2f0f6ec3f1c6c00e2fca949996c7e154a760f6eedc8: Status 404 returned error can't find the container with id 997f78fa735b62216b42b2f0f6ec3f1c6c00e2fca949996c7e154a760f6eedc8 Jul 21 04:37:01.871982 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.871943 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:37:01.872093 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.872026 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:37:01.872093 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:01.872072 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:37:02.380583 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:02.380541 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" event={"ID":"c09e9457-109b-44f2-8a49-3dc5d72c45ea","Type":"ContainerStarted","Data":"a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a"} Jul 21 04:37:02.380801 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:02.380598 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" event={"ID":"c09e9457-109b-44f2-8a49-3dc5d72c45ea","Type":"ContainerStarted","Data":"997f78fa735b62216b42b2f0f6ec3f1c6c00e2fca949996c7e154a760f6eedc8"} Jul 21 04:37:02.403163 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:02.403104 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" podStartSLOduration=1.4030901949999999 podStartE2EDuration="1.403090195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:37:02.40164427 +0000 UTC m=+1667.800977027" watchObservedRunningTime="2026-07-21 04:37:02.403090195 +0000 UTC m=+1667.802422953" Jul 21 04:37:02.723264 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:02.723170 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:02.728429 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:02.728403 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:03.385604 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:03.385568 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:03.386750 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:03.386727 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:05.644462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.644398 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:37:05.726640 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.726606 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrqxg\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-kube-api-access-hrqxg\") pod \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " Jul 21 04:37:05.726810 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.726660 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-socket\") pod \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " Jul 21 04:37:05.726810 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.726694 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-envoy\") pod \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " Jul 21 04:37:05.726810 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.726773 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-credential-socket\") pod \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " Jul 21 04:37:05.726810 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.726802 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istiod-ca-cert\") pod \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " Jul 21 04:37:05.727015 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.726830 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-podinfo\") pod \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " Jul 21 04:37:05.727015 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.726852 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-data\") pod \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " Jul 21 04:37:05.727015 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.726895 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-token\") pod \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " Jul 21 04:37:05.727015 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.726936 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-certs\") pod \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\" (UID: \"3a5c6e12-cc79-4d84-a652-c3c12fe03027\") " Jul 21 04:37:05.727222 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.727048 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3a5c6e12-cc79-4d84-a652-c3c12fe03027" (UID: "3a5c6e12-cc79-4d84-a652-c3c12fe03027"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:05.727222 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.727167 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3a5c6e12-cc79-4d84-a652-c3c12fe03027" (UID: "3a5c6e12-cc79-4d84-a652-c3c12fe03027"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:37:05.727386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.727261 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3a5c6e12-cc79-4d84-a652-c3c12fe03027" (UID: "3a5c6e12-cc79-4d84-a652-c3c12fe03027"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:05.727386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.727320 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3a5c6e12-cc79-4d84-a652-c3c12fe03027" (UID: "3a5c6e12-cc79-4d84-a652-c3c12fe03027"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:05.727477 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.727449 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-credential-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:05.727477 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.727467 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istiod-ca-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:05.727635 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.727482 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:05.727635 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.727495 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-workload-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:05.728208 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.728166 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-data" (OuterVolumeSpecName: "istio-data") pod "3a5c6e12-cc79-4d84-a652-c3c12fe03027" (UID: "3a5c6e12-cc79-4d84-a652-c3c12fe03027"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:05.729086 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.729059 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-kube-api-access-hrqxg" (OuterVolumeSpecName: "kube-api-access-hrqxg") pod "3a5c6e12-cc79-4d84-a652-c3c12fe03027" (UID: "3a5c6e12-cc79-4d84-a652-c3c12fe03027"). InnerVolumeSpecName "kube-api-access-hrqxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:37:05.729505 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.729476 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3a5c6e12-cc79-4d84-a652-c3c12fe03027" (UID: "3a5c6e12-cc79-4d84-a652-c3c12fe03027"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 04:37:05.729620 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.729600 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a5c6e12-cc79-4d84-a652-c3c12fe03027" (UID: "3a5c6e12-cc79-4d84-a652-c3c12fe03027"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:37:05.729738 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.729715 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3a5c6e12-cc79-4d84-a652-c3c12fe03027" (UID: "3a5c6e12-cc79-4d84-a652-c3c12fe03027"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:05.828200 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.828170 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrqxg\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-kube-api-access-hrqxg\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:05.828200 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.828195 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-envoy\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:05.828200 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.828205 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-podinfo\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:05.828528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.828213 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-data\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:05.828528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:05.828222 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5c6e12-cc79-4d84-a652-c3c12fe03027-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:06.404546 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.404515 2567 generic.go:356] "Generic (PLEG): container finished" podID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerID="f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c" exitCode=0 Jul 21 04:37:06.404732 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.404563 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" Jul 21 04:37:06.404732 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.404602 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" event={"ID":"3a5c6e12-cc79-4d84-a652-c3c12fe03027","Type":"ContainerDied","Data":"f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c"} Jul 21 04:37:06.404732 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.404668 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt" event={"ID":"3a5c6e12-cc79-4d84-a652-c3c12fe03027","Type":"ContainerDied","Data":"a8e09d5bc7446fb48e5f14484c19239b234ce7022e9c032af1846a74f21363ad"} Jul 21 04:37:06.404732 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.404692 2567 scope.go:117] "RemoveContainer" containerID="f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c" Jul 21 04:37:06.415745 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.415679 2567 scope.go:117] "RemoveContainer" containerID="6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448" Jul 21 04:37:06.427881 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.427858 2567 scope.go:117] "RemoveContainer" containerID="f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c" Jul 21 04:37:06.428252 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:37:06.428218 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c\": container with ID starting with f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c not found: ID does not exist" containerID="f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c" Jul 21 04:37:06.428394 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.428264 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c"} err="failed to get container status \"f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c\": rpc error: code = NotFound desc = could not find container \"f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c\": container with ID starting with f0e0ff57e9a315dfbfb55bd95477250b6d797ecf8ee1891772bbce2e5aff7c6c not found: ID does not exist" Jul 21 04:37:06.428394 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.428291 2567 scope.go:117] "RemoveContainer" containerID="6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448" Jul 21 04:37:06.428567 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:37:06.428533 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448\": container with ID starting with 6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448 not found: ID does not exist" containerID="6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448" Jul 21 04:37:06.428660 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.428573 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448"} err="failed to get container status \"6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448\": rpc error: code = NotFound desc = could not find container \"6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448\": container with ID starting with 6536fe64e17a17f74a0ea963e3d14c03d1d0274c366476e820f6135019378448 not found: ID does not exist" Jul 21 04:37:06.435017 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.434995 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt"] Jul 21 04:37:06.441837 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:06.441816 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fc7e6220-openshift-default-56855bf48f-tl2dt"] Jul 21 04:37:07.267227 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:07.267195 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" path="/var/lib/kubelet/pods/3a5c6e12-cc79-4d84-a652-c3c12fe03027/volumes" Jul 21 04:37:09.979484 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:09.979451 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:37:18.387670 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:18.387633 2567 patch_prober.go:28] interesting pod/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 21 04:37:18.388107 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:18.387720 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 21 04:37:18.489843 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:18.489806 2567 generic.go:356] "Generic (PLEG): container finished" podID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerID="a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a" exitCode=0 Jul 21 04:37:18.490021 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:18.489875 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" event={"ID":"c09e9457-109b-44f2-8a49-3dc5d72c45ea","Type":"ContainerDied","Data":"a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a"} Jul 21 04:37:18.490267 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:18.490254 2567 scope.go:117] "RemoveContainer" containerID="a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a" Jul 21 04:37:18.491280 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:18.491251 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:37:18.491349 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:18.491319 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:37:18.491399 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:18.491366 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:37:19.497049 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:19.497016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" event={"ID":"c09e9457-109b-44f2-8a49-3dc5d72c45ea","Type":"ContainerStarted","Data":"64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad"} Jul 21 04:37:19.723444 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:19.723400 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:19.723694 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:19.723671 2567 patch_prober.go:28] interesting pod/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 21 04:37:19.723740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:19.723723 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 21 04:37:20.723680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:20.723631 2567 patch_prober.go:28] interesting pod/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 21 04:37:20.724199 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:20.723703 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 21 04:37:21.723937 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:21.723895 2567 patch_prober.go:28] interesting pod/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 21 04:37:21.724582 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:21.724547 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 21 04:37:22.723708 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:22.723664 2567 patch_prober.go:28] interesting pod/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 21 04:37:22.723997 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:22.723965 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 21 04:37:23.723465 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:23.723428 2567 patch_prober.go:28] interesting pod/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 21 04:37:23.723763 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:23.723737 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 21 04:37:24.726555 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:24.726525 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:24.727033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:24.726776 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:24.727561 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:24.727543 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:42.587681 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:42.587584 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:37:44.151577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:44.151539 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn"] Jul 21 04:37:44.151967 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:44.151752 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" containerID="cri-o://64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad" gracePeriod=30 Jul 21 04:37:49.402969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.402946 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:49.522110 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522062 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-socket\") pod \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " Jul 21 04:37:49.522312 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522128 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-envoy\") pod \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " Jul 21 04:37:49.522312 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522202 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-token\") pod \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " Jul 21 04:37:49.522312 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522223 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-credential-socket\") pod \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " Jul 21 04:37:49.522312 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522243 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-data\") pod \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " Jul 21 04:37:49.522577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522423 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2mx5\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-kube-api-access-n2mx5\") pod \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " Jul 21 04:37:49.522577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522496 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-certs\") pod \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " Jul 21 04:37:49.522577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522540 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istiod-ca-cert\") pod \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " Jul 21 04:37:49.522577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522570 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-podinfo\") pod \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\" (UID: \"c09e9457-109b-44f2-8a49-3dc5d72c45ea\") " Jul 21 04:37:49.523179 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522422 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c09e9457-109b-44f2-8a49-3dc5d72c45ea" (UID: "c09e9457-109b-44f2-8a49-3dc5d72c45ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:49.523324 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522632 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c09e9457-109b-44f2-8a49-3dc5d72c45ea" (UID: "c09e9457-109b-44f2-8a49-3dc5d72c45ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:49.523324 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522776 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c09e9457-109b-44f2-8a49-3dc5d72c45ea" (UID: "c09e9457-109b-44f2-8a49-3dc5d72c45ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:49.523324 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.522955 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c09e9457-109b-44f2-8a49-3dc5d72c45ea" (UID: "c09e9457-109b-44f2-8a49-3dc5d72c45ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:37:49.523324 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.523222 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "c09e9457-109b-44f2-8a49-3dc5d72c45ea" (UID: "c09e9457-109b-44f2-8a49-3dc5d72c45ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:49.525060 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.525034 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-kube-api-access-n2mx5" (OuterVolumeSpecName: "kube-api-access-n2mx5") pod "c09e9457-109b-44f2-8a49-3dc5d72c45ea" (UID: "c09e9457-109b-44f2-8a49-3dc5d72c45ea"). InnerVolumeSpecName "kube-api-access-n2mx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:37:49.525060 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.525205 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c09e9457-109b-44f2-8a49-3dc5d72c45ea" (UID: "c09e9457-109b-44f2-8a49-3dc5d72c45ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:37:49.525442 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.525397 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c09e9457-109b-44f2-8a49-3dc5d72c45ea" (UID: "c09e9457-109b-44f2-8a49-3dc5d72c45ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 04:37:49.525607 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.525590 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "c09e9457-109b-44f2-8a49-3dc5d72c45ea" (UID: "c09e9457-109b-44f2-8a49-3dc5d72c45ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:37:49.624330 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.624299 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:49.624330 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.624328 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-credential-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:49.624330 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.624352 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-data\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:49.624559 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.624361 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2mx5\" (UniqueName: \"kubernetes.io/projected/c09e9457-109b-44f2-8a49-3dc5d72c45ea-kube-api-access-n2mx5\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:49.624559 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.624371 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:49.624559 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.624380 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istiod-ca-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:49.624559 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.624389 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-podinfo\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:49.624559 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.624397 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-workload-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:49.624559 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.624406 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09e9457-109b-44f2-8a49-3dc5d72c45ea-istio-envoy\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:37:49.665131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.665100 2567 generic.go:356] "Generic (PLEG): container finished" podID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerID="64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad" exitCode=0 Jul 21 04:37:49.665281 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.665145 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" Jul 21 04:37:49.665281 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.665142 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" event={"ID":"c09e9457-109b-44f2-8a49-3dc5d72c45ea","Type":"ContainerDied","Data":"64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad"} Jul 21 04:37:49.665281 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.665258 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn" event={"ID":"c09e9457-109b-44f2-8a49-3dc5d72c45ea","Type":"ContainerDied","Data":"997f78fa735b62216b42b2f0f6ec3f1c6c00e2fca949996c7e154a760f6eedc8"} Jul 21 04:37:49.665424 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.665285 2567 scope.go:117] "RemoveContainer" containerID="64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad" Jul 21 04:37:49.675160 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.675142 2567 scope.go:117] "RemoveContainer" containerID="a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a" Jul 21 04:37:49.683576 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.683558 2567 scope.go:117] "RemoveContainer" containerID="64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad" Jul 21 04:37:49.683846 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:37:49.683825 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad\": container with ID starting with 64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad not found: ID does not exist" containerID="64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad" Jul 21 04:37:49.683900 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.683853 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad"} err="failed to get container status \"64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad\": rpc error: code = NotFound desc = could not find container \"64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad\": container with ID starting with 64f3485aa647d81a5d7bd91128d1ae4fdf7dfffba2ee2c0d86ba8fb142c1f2ad not found: ID does not exist" Jul 21 04:37:49.683900 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.683875 2567 scope.go:117] "RemoveContainer" containerID="a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a" Jul 21 04:37:49.684058 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:37:49.684043 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a\": container with ID starting with a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a not found: ID does not exist" containerID="a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a" Jul 21 04:37:49.684098 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.684062 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a"} err="failed to get container status \"a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a\": rpc error: code = NotFound desc = could not find container \"a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a\": container with ID starting with a9fb62b6a52e30fbab0468dd6de268eb461dc4b921c3870de293a236ecdd048a not found: ID does not exist" Jul 21 04:37:49.689375 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.689331 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn"] Jul 21 04:37:49.691566 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:49.691546 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c0006c3d-gw-openshift-default-55b8848b9f-xb4dn"] Jul 21 04:37:51.266919 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:51.266888 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" path="/var/lib/kubelet/pods/c09e9457-109b-44f2-8a49-3dc5d72c45ea/volumes" Jul 21 04:37:53.889231 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:37:53.889181 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:38:27.080560 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:27.080521 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:38:30.738979 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.738944 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87"] Jul 21 04:38:30.739384 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.739368 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" Jul 21 04:38:30.739384 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.739380 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" Jul 21 04:38:30.739464 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.739393 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" Jul 21 04:38:30.739464 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.739399 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" Jul 21 04:38:30.739464 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.739416 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" Jul 21 04:38:30.739464 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.739423 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" Jul 21 04:38:30.739591 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.739485 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" Jul 21 04:38:30.739591 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.739493 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" Jul 21 04:38:30.739591 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.739502 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" Jul 21 04:38:30.742534 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.742517 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.744958 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.744936 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-dfb01d92-openshift-default-dockercfg-ktrlg\"" Jul 21 04:38:30.754432 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.754411 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87"] Jul 21 04:38:30.858669 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.858632 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-credential-socket\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.858669 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.858674 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvnr8\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-kube-api-access-tvnr8\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.858910 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.858698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-podinfo\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.858910 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.858780 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-data\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.858910 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.858809 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-token\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.858910 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.858846 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246fe568-59cd-4e42-a70e-4e2898bd5a36-istiod-ca-cert\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.859040 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.858919 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-envoy\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.859040 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.858962 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-socket\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.859040 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.858988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-certs\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960237 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-data\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960237 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-token\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960263 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246fe568-59cd-4e42-a70e-4e2898bd5a36-istiod-ca-cert\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-envoy\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-socket\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960390 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-certs\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-credential-socket\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960480 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvnr8\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-kube-api-access-tvnr8\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960512 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-podinfo\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960905 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960715 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-data\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.960905 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960862 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-credential-socket\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.961026 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.960930 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-certs\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.961083 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.961055 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-socket\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.961164 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.961138 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246fe568-59cd-4e42-a70e-4e2898bd5a36-istiod-ca-cert\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.962701 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.962680 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-envoy\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.962916 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.962899 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-podinfo\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.969295 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.969274 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvnr8\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-kube-api-access-tvnr8\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:30.969563 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:30.969543 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-token\") pod \"e2e-derive-dfb01d92-openshift-default-59c67679-7cb87\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:31.052431 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:31.052397 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:31.197525 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:31.197497 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87"] Jul 21 04:38:31.199619 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:38:31.199586 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246fe568_59cd_4e42_a70e_4e2898bd5a36.slice/crio-fa77767977a421857bd4642d43e1a3b705860e5c881555265d72afcb92973951 WatchSource:0}: Error finding container fa77767977a421857bd4642d43e1a3b705860e5c881555265d72afcb92973951: Status 404 returned error can't find the container with id fa77767977a421857bd4642d43e1a3b705860e5c881555265d72afcb92973951 Jul 21 04:38:31.202060 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:31.202017 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:38:31.202153 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:31.202115 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:38:31.202242 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:31.202159 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:38:31.895005 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:31.894962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" event={"ID":"246fe568-59cd-4e42-a70e-4e2898bd5a36","Type":"ContainerStarted","Data":"fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22"} Jul 21 04:38:31.895498 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:31.895024 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" event={"ID":"246fe568-59cd-4e42-a70e-4e2898bd5a36","Type":"ContainerStarted","Data":"fa77767977a421857bd4642d43e1a3b705860e5c881555265d72afcb92973951"} Jul 21 04:38:31.915371 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:31.915297 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" podStartSLOduration=1.91527639 podStartE2EDuration="1.91527639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:38:31.914504412 +0000 UTC m=+1757.313837198" watchObservedRunningTime="2026-07-21 04:38:31.91527639 +0000 UTC m=+1757.314609137" Jul 21 04:38:32.052991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:32.052955 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:32.058273 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:32.058246 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:32.900302 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:32.900269 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:32.901370 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:32.901331 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:38.373692 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:38.373661 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:38:46.984196 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:46.984162 2567 generic.go:356] "Generic (PLEG): container finished" podID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerID="fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22" exitCode=0 Jul 21 04:38:46.984674 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:46.984244 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" event={"ID":"246fe568-59cd-4e42-a70e-4e2898bd5a36","Type":"ContainerDied","Data":"fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22"} Jul 21 04:38:46.984771 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:46.984740 2567 scope.go:117] "RemoveContainer" containerID="fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22" Jul 21 04:38:46.985801 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:46.985767 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:38:46.985870 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:46.985847 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:38:46.985907 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:46.985883 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:38:47.900417 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:47.900380 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:47.991919 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:47.991884 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" event={"ID":"246fe568-59cd-4e42-a70e-4e2898bd5a36","Type":"ContainerStarted","Data":"d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd"} Jul 21 04:38:48.053504 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:48.053463 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:48.054773 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:48.054746 2567 patch_prober.go:28] interesting pod/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 21 04:38:48.054927 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:48.054816 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 21 04:38:49.053253 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:49.053216 2567 patch_prober.go:28] interesting pod/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 21 04:38:49.053685 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:49.053270 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 21 04:38:50.053082 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:50.053048 2567 patch_prober.go:28] interesting pod/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 21 04:38:50.053253 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:50.053101 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 21 04:38:51.053511 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:51.053474 2567 patch_prober.go:28] interesting pod/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 21 04:38:51.053903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:51.053528 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 21 04:38:52.057626 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:52.057597 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:52.058007 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:52.057952 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:38:52.058577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:38:52.058560 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:39:11.771934 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:11.771898 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:39:13.441757 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:13.441722 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87"] Jul 21 04:39:13.442235 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:13.441980 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" containerID="cri-o://d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd" gracePeriod=30 Jul 21 04:39:15.186823 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:15.186794 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:39:15.190877 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:15.190853 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:39:18.696729 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.696705 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:39:18.806481 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806448 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-data\") pod \"246fe568-59cd-4e42-a70e-4e2898bd5a36\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " Jul 21 04:39:18.806679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806517 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-envoy\") pod \"246fe568-59cd-4e42-a70e-4e2898bd5a36\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " Jul 21 04:39:18.806679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806542 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-credential-socket\") pod \"246fe568-59cd-4e42-a70e-4e2898bd5a36\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " Jul 21 04:39:18.806679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806581 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-certs\") pod \"246fe568-59cd-4e42-a70e-4e2898bd5a36\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " Jul 21 04:39:18.806679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806616 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-podinfo\") pod \"246fe568-59cd-4e42-a70e-4e2898bd5a36\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " Jul 21 04:39:18.806679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806637 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-token\") pod \"246fe568-59cd-4e42-a70e-4e2898bd5a36\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " Jul 21 04:39:18.806973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806699 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-socket\") pod \"246fe568-59cd-4e42-a70e-4e2898bd5a36\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " Jul 21 04:39:18.806973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806730 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvnr8\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-kube-api-access-tvnr8\") pod \"246fe568-59cd-4e42-a70e-4e2898bd5a36\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " Jul 21 04:39:18.806973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806769 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246fe568-59cd-4e42-a70e-4e2898bd5a36-istiod-ca-cert\") pod \"246fe568-59cd-4e42-a70e-4e2898bd5a36\" (UID: \"246fe568-59cd-4e42-a70e-4e2898bd5a36\") " Jul 21 04:39:18.806973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806887 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "246fe568-59cd-4e42-a70e-4e2898bd5a36" (UID: "246fe568-59cd-4e42-a70e-4e2898bd5a36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:39:18.806973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.806898 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "246fe568-59cd-4e42-a70e-4e2898bd5a36" (UID: "246fe568-59cd-4e42-a70e-4e2898bd5a36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:39:18.807220 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.807097 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "246fe568-59cd-4e42-a70e-4e2898bd5a36" (UID: "246fe568-59cd-4e42-a70e-4e2898bd5a36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:39:18.807220 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.807152 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-credential-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:39:18.807220 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.807172 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:39:18.807373 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.807264 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246fe568-59cd-4e42-a70e-4e2898bd5a36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "246fe568-59cd-4e42-a70e-4e2898bd5a36" (UID: "246fe568-59cd-4e42-a70e-4e2898bd5a36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:39:18.807511 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.807486 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-data" (OuterVolumeSpecName: "istio-data") pod "246fe568-59cd-4e42-a70e-4e2898bd5a36" (UID: "246fe568-59cd-4e42-a70e-4e2898bd5a36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:39:18.808737 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.808710 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "246fe568-59cd-4e42-a70e-4e2898bd5a36" (UID: "246fe568-59cd-4e42-a70e-4e2898bd5a36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:39:18.808882 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.808865 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "246fe568-59cd-4e42-a70e-4e2898bd5a36" (UID: "246fe568-59cd-4e42-a70e-4e2898bd5a36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 04:39:18.809178 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.809163 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-kube-api-access-tvnr8" (OuterVolumeSpecName: "kube-api-access-tvnr8") pod "246fe568-59cd-4e42-a70e-4e2898bd5a36" (UID: "246fe568-59cd-4e42-a70e-4e2898bd5a36"). InnerVolumeSpecName "kube-api-access-tvnr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:39:18.809240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.809182 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-token" (OuterVolumeSpecName: "istio-token") pod "246fe568-59cd-4e42-a70e-4e2898bd5a36" (UID: "246fe568-59cd-4e42-a70e-4e2898bd5a36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:39:18.908223 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.908186 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-data\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:39:18.908223 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.908222 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-envoy\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:39:18.908454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.908235 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-podinfo\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:39:18.908454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.908249 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:39:18.908454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.908264 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/246fe568-59cd-4e42-a70e-4e2898bd5a36-workload-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:39:18.908454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.908278 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvnr8\" (UniqueName: \"kubernetes.io/projected/246fe568-59cd-4e42-a70e-4e2898bd5a36-kube-api-access-tvnr8\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:39:18.908454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:18.908294 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/246fe568-59cd-4e42-a70e-4e2898bd5a36-istiod-ca-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:39:19.174976 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.174893 2567 generic.go:356] "Generic (PLEG): container finished" podID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerID="d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd" exitCode=0 Jul 21 04:39:19.174976 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.174944 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" Jul 21 04:39:19.175153 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.174968 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" event={"ID":"246fe568-59cd-4e42-a70e-4e2898bd5a36","Type":"ContainerDied","Data":"d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd"} Jul 21 04:39:19.175153 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.175028 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87" event={"ID":"246fe568-59cd-4e42-a70e-4e2898bd5a36","Type":"ContainerDied","Data":"fa77767977a421857bd4642d43e1a3b705860e5c881555265d72afcb92973951"} Jul 21 04:39:19.175153 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.175055 2567 scope.go:117] "RemoveContainer" containerID="d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd" Jul 21 04:39:19.184569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.184552 2567 scope.go:117] "RemoveContainer" containerID="fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22" Jul 21 04:39:19.192635 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.192618 2567 scope.go:117] "RemoveContainer" containerID="d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd" Jul 21 04:39:19.192890 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:39:19.192871 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd\": container with ID starting with d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd not found: ID does not exist" containerID="d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd" Jul 21 04:39:19.192951 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.192900 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd"} err="failed to get container status \"d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd\": rpc error: code = NotFound desc = could not find container \"d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd\": container with ID starting with d6730ba320e52eaca937c210438e45d6b9063fa9a9f06df7da3adfafb3b706bd not found: ID does not exist" Jul 21 04:39:19.192951 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.192917 2567 scope.go:117] "RemoveContainer" containerID="fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22" Jul 21 04:39:19.193143 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:39:19.193126 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22\": container with ID starting with fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22 not found: ID does not exist" containerID="fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22" Jul 21 04:39:19.193182 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.193149 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22"} err="failed to get container status \"fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22\": rpc error: code = NotFound desc = could not find container \"fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22\": container with ID starting with fbb125505d821cbc0155bcc2b36de6f92303324a914c3cc409cebd975c4f1b22 not found: ID does not exist" Jul 21 04:39:19.200321 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.200301 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87"] Jul 21 04:39:19.204609 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.204590 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-dfb01d92-openshift-default-59c67679-7cb87"] Jul 21 04:39:19.267241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:39:19.267209 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" path="/var/lib/kubelet/pods/246fe568-59cd-4e42-a70e-4e2898bd5a36/volumes" Jul 21 04:40:44.475734 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:44.475648 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:40:56.390935 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:56.390897 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:40:57.740274 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.740240 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78"] Jul 21 04:40:57.740677 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.740636 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" Jul 21 04:40:57.740677 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.740650 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5c6e12-cc79-4d84-a652-c3c12fe03027" containerName="istio-proxy" Jul 21 04:40:57.740677 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.740667 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" Jul 21 04:40:57.740677 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.740673 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" Jul 21 04:40:57.740825 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.740696 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" Jul 21 04:40:57.740825 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.740701 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" Jul 21 04:40:57.740825 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.740764 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" Jul 21 04:40:57.740825 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.740774 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c09e9457-109b-44f2-8a49-3dc5d72c45ea" containerName="istio-proxy" Jul 21 04:40:57.743860 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.743842 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.746367 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.746322 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6f1bb607-openshift-default-dockercfg-vmkfg\"" Jul 21 04:40:57.768625 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.768597 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78"] Jul 21 04:40:57.896792 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.896752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-data\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.897003 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.896803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370dfb47-2616-49d4-938d-a557ca9e914e-istio-podinfo\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.897003 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.896848 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-istio-token\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.897003 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.896911 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-socket\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.897003 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.896963 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-certs\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.897179 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.897045 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370dfb47-2616-49d4-938d-a557ca9e914e-istiod-ca-cert\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.897179 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.897091 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrxq\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-kube-api-access-fwrxq\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.897179 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.897115 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-envoy\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.897286 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.897245 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-credential-socket\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.998758 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.998666 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-credential-socket\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.998758 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.998707 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-data\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.998758 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.998724 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370dfb47-2616-49d4-938d-a557ca9e914e-istio-podinfo\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.998758 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.998749 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-istio-token\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.998769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-socket\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.998787 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-certs\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.998819 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370dfb47-2616-49d4-938d-a557ca9e914e-istiod-ca-cert\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.998857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrxq\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-kube-api-access-fwrxq\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.998875 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-envoy\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999469 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.999093 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-data\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999469 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.999368 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-credential-socket\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999469 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.999372 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-socket\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999469 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.999423 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-certs\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:57.999722 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:57.999697 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370dfb47-2616-49d4-938d-a557ca9e914e-istiod-ca-cert\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:58.001200 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.001183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-envoy\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:58.001486 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.001458 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370dfb47-2616-49d4-938d-a557ca9e914e-istio-podinfo\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:58.008010 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.007984 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrxq\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-kube-api-access-fwrxq\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:58.008101 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.008034 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-istio-token\") pod \"e2e-mt-6f1bb607-openshift-default-586b95d879-55g78\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:58.054990 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.054956 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:58.205474 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.205448 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78"] Jul 21 04:40:58.207790 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:40:58.207763 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370dfb47_2616_49d4_938d_a557ca9e914e.slice/crio-1476303cbec94ed6a88f0b1b9063171f4d45f754bc3b5c2661e44b06ef28b54e WatchSource:0}: Error finding container 1476303cbec94ed6a88f0b1b9063171f4d45f754bc3b5c2661e44b06ef28b54e: Status 404 returned error can't find the container with id 1476303cbec94ed6a88f0b1b9063171f4d45f754bc3b5c2661e44b06ef28b54e Jul 21 04:40:58.209979 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.209942 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:40:58.210071 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.210008 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:40:58.210071 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.210036 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:40:58.733467 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.733432 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" event={"ID":"370dfb47-2616-49d4-938d-a557ca9e914e","Type":"ContainerStarted","Data":"4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133"} Jul 21 04:40:58.733631 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.733484 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" event={"ID":"370dfb47-2616-49d4-938d-a557ca9e914e","Type":"ContainerStarted","Data":"1476303cbec94ed6a88f0b1b9063171f4d45f754bc3b5c2661e44b06ef28b54e"} Jul 21 04:40:58.756501 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:58.756450 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" podStartSLOduration=1.756433451 podStartE2EDuration="1.756433451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:40:58.755003207 +0000 UTC m=+1904.154335965" watchObservedRunningTime="2026-07-21 04:40:58.756433451 +0000 UTC m=+1904.155766230" Jul 21 04:40:59.055577 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:59.055543 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:59.060745 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:59.060720 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:59.739312 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:59.739283 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:40:59.740244 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:40:59.740224 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:41:06.577916 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:06.577883 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:41:14.740610 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:14.740525 2567 patch_prober.go:28] interesting pod/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 21 04:41:14.740610 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:14.740579 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 21 04:41:14.824801 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:14.824763 2567 generic.go:356] "Generic (PLEG): container finished" podID="370dfb47-2616-49d4-938d-a557ca9e914e" containerID="4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133" exitCode=0 Jul 21 04:41:14.824963 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:14.824832 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" event={"ID":"370dfb47-2616-49d4-938d-a557ca9e914e","Type":"ContainerDied","Data":"4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133"} Jul 21 04:41:14.825293 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:14.825273 2567 scope.go:117] "RemoveContainer" containerID="4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133" Jul 21 04:41:14.826377 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:14.826326 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:41:14.826463 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:14.826415 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:41:14.826463 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:14.826448 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:41:15.832508 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:15.832475 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" event={"ID":"370dfb47-2616-49d4-938d-a557ca9e914e","Type":"ContainerStarted","Data":"7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526"} Jul 21 04:41:16.055207 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:16.055164 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:41:16.055462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:16.055433 2567 patch_prober.go:28] interesting pod/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 21 04:41:16.055639 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:16.055491 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 21 04:41:17.056245 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:17.056202 2567 patch_prober.go:28] interesting pod/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 21 04:41:17.056757 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:17.056278 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 21 04:41:18.056223 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:18.056181 2567 patch_prober.go:28] interesting pod/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 21 04:41:18.056472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:18.056243 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 21 04:41:19.056166 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:19.056130 2567 patch_prober.go:28] interesting pod/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 21 04:41:19.056368 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:19.056184 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 21 04:41:20.055662 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:20.055618 2567 patch_prober.go:28] interesting pod/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 21 04:41:20.056154 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:20.055689 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 21 04:41:21.060839 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:21.060803 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:41:21.061287 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:21.061112 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:41:21.061848 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:21.061830 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:41:26.077259 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:26.077220 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:41:34.259087 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:34.259053 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78"] Jul 21 04:41:34.259646 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:34.259303 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" containerID="cri-o://7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526" gracePeriod=30 Jul 21 04:41:36.063550 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:36.063507 2567 patch_prober.go:28] interesting pod/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 04:41:36.063956 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:36.063583 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 04:41:39.524094 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.524027 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:41:39.677719 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.677680 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370dfb47-2616-49d4-938d-a557ca9e914e-istio-podinfo\") pod \"370dfb47-2616-49d4-938d-a557ca9e914e\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " Jul 21 04:41:39.677719 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.677722 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-certs\") pod \"370dfb47-2616-49d4-938d-a557ca9e914e\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " Jul 21 04:41:39.677968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.677743 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwrxq\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-kube-api-access-fwrxq\") pod \"370dfb47-2616-49d4-938d-a557ca9e914e\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " Jul 21 04:41:39.677968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.677804 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-data\") pod \"370dfb47-2616-49d4-938d-a557ca9e914e\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " Jul 21 04:41:39.677968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.677820 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-socket\") pod \"370dfb47-2616-49d4-938d-a557ca9e914e\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " Jul 21 04:41:39.677968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.677840 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-envoy\") pod \"370dfb47-2616-49d4-938d-a557ca9e914e\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " Jul 21 04:41:39.677968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.677891 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-istio-token\") pod \"370dfb47-2616-49d4-938d-a557ca9e914e\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " Jul 21 04:41:39.677968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.677929 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370dfb47-2616-49d4-938d-a557ca9e914e-istiod-ca-cert\") pod \"370dfb47-2616-49d4-938d-a557ca9e914e\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " Jul 21 04:41:39.677968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.677954 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-credential-socket\") pod \"370dfb47-2616-49d4-938d-a557ca9e914e\" (UID: \"370dfb47-2616-49d4-938d-a557ca9e914e\") " Jul 21 04:41:39.678369 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.678103 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "370dfb47-2616-49d4-938d-a557ca9e914e" (UID: "370dfb47-2616-49d4-938d-a557ca9e914e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:41:39.678369 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.678163 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "370dfb47-2616-49d4-938d-a557ca9e914e" (UID: "370dfb47-2616-49d4-938d-a557ca9e914e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:41:39.678369 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.678304 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:41:39.678540 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.678386 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-workload-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:41:39.678540 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.678491 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370dfb47-2616-49d4-938d-a557ca9e914e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "370dfb47-2616-49d4-938d-a557ca9e914e" (UID: "370dfb47-2616-49d4-938d-a557ca9e914e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:41:39.678655 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.678587 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "370dfb47-2616-49d4-938d-a557ca9e914e" (UID: "370dfb47-2616-49d4-938d-a557ca9e914e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:41:39.678805 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.678782 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-data" (OuterVolumeSpecName: "istio-data") pod "370dfb47-2616-49d4-938d-a557ca9e914e" (UID: "370dfb47-2616-49d4-938d-a557ca9e914e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:41:39.680137 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.680096 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "370dfb47-2616-49d4-938d-a557ca9e914e" (UID: "370dfb47-2616-49d4-938d-a557ca9e914e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:41:39.680238 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.680170 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-kube-api-access-fwrxq" (OuterVolumeSpecName: "kube-api-access-fwrxq") pod "370dfb47-2616-49d4-938d-a557ca9e914e" (UID: "370dfb47-2616-49d4-938d-a557ca9e914e"). InnerVolumeSpecName "kube-api-access-fwrxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:41:39.680367 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.680333 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/370dfb47-2616-49d4-938d-a557ca9e914e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "370dfb47-2616-49d4-938d-a557ca9e914e" (UID: "370dfb47-2616-49d4-938d-a557ca9e914e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 04:41:39.680472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.680454 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-istio-token" (OuterVolumeSpecName: "istio-token") pod "370dfb47-2616-49d4-938d-a557ca9e914e" (UID: "370dfb47-2616-49d4-938d-a557ca9e914e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:41:39.779131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.779039 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-data\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:41:39.779131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.779080 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-istio-envoy\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:41:39.779131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.779090 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:41:39.779131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.779100 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/370dfb47-2616-49d4-938d-a557ca9e914e-istiod-ca-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:41:39.779131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.779108 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/370dfb47-2616-49d4-938d-a557ca9e914e-credential-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:41:39.779131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.779117 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/370dfb47-2616-49d4-938d-a557ca9e914e-istio-podinfo\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:41:39.779131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.779125 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrxq\" (UniqueName: \"kubernetes.io/projected/370dfb47-2616-49d4-938d-a557ca9e914e-kube-api-access-fwrxq\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:41:39.980052 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.980016 2567 generic.go:356] "Generic (PLEG): container finished" podID="370dfb47-2616-49d4-938d-a557ca9e914e" containerID="7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526" exitCode=0 Jul 21 04:41:39.980221 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.980071 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" Jul 21 04:41:39.980221 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.980087 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" event={"ID":"370dfb47-2616-49d4-938d-a557ca9e914e","Type":"ContainerDied","Data":"7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526"} Jul 21 04:41:39.980221 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.980144 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78" event={"ID":"370dfb47-2616-49d4-938d-a557ca9e914e","Type":"ContainerDied","Data":"1476303cbec94ed6a88f0b1b9063171f4d45f754bc3b5c2661e44b06ef28b54e"} Jul 21 04:41:39.980221 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.980174 2567 scope.go:117] "RemoveContainer" containerID="7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526" Jul 21 04:41:39.990054 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.990034 2567 scope.go:117] "RemoveContainer" containerID="4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133" Jul 21 04:41:39.998182 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.998168 2567 scope.go:117] "RemoveContainer" containerID="7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526" Jul 21 04:41:39.998511 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:41:39.998487 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526\": container with ID starting with 7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526 not found: ID does not exist" containerID="7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526" Jul 21 04:41:39.998604 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.998518 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526"} err="failed to get container status \"7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526\": rpc error: code = NotFound desc = could not find container \"7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526\": container with ID starting with 7c256ce3eb70a1def45a6c9cf36127817e5133ca9bd520022f6f2794b11fd526 not found: ID does not exist" Jul 21 04:41:39.998604 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.998535 2567 scope.go:117] "RemoveContainer" containerID="4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133" Jul 21 04:41:39.998788 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:41:39.998771 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133\": container with ID starting with 4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133 not found: ID does not exist" containerID="4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133" Jul 21 04:41:39.998825 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:39.998792 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133"} err="failed to get container status \"4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133\": rpc error: code = NotFound desc = could not find container \"4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133\": container with ID starting with 4215fa06c5d72e3292e50960cd84887002506d3305dc165d7622cd6c3b7e2133 not found: ID does not exist" Jul 21 04:41:40.003714 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:40.003692 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78"] Jul 21 04:41:40.008062 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:40.008040 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6f1bb607-openshift-default-586b95d879-55g78"] Jul 21 04:41:41.274049 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:41.274014 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" path="/var/lib/kubelet/pods/370dfb47-2616-49d4-938d-a557ca9e914e/volumes" Jul 21 04:41:54.345273 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.345238 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj"] Jul 21 04:41:54.345873 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.345851 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" Jul 21 04:41:54.345935 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.345875 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" Jul 21 04:41:54.345935 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.345894 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" Jul 21 04:41:54.345935 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.345900 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" Jul 21 04:41:54.346074 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.346007 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" Jul 21 04:41:54.346074 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.346019 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="246fe568-59cd-4e42-a70e-4e2898bd5a36" containerName="istio-proxy" Jul 21 04:41:54.349037 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.349018 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.351788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.351766 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6ae222fa-openshift-default-dockercfg-d8s5g\"" Jul 21 04:41:54.359938 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.359914 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj"] Jul 21 04:41:54.413418 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.413385 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-data\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.413418 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.413415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-podinfo\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.413630 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.413436 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-token\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.413630 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.413525 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-envoy\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.413630 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.413544 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-credential-socket\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.413630 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.413559 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4fc9\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-kube-api-access-h4fc9\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.413630 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.413626 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-socket\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.413822 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.413649 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-certs\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.413822 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.413668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d25a72d-e345-4927-b69a-ecb298afa2f4-istiod-ca-cert\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514243 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514211 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-socket\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514252 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-certs\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514302 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d25a72d-e345-4927-b69a-ecb298afa2f4-istiod-ca-cert\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514433 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514410 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-data\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-podinfo\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-token\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-envoy\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514550 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-credential-socket\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514605 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4fc9\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-kube-api-access-h4fc9\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514863 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514715 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-socket\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514920 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514889 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-data\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.514983 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.514965 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-credential-socket\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.515195 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.515174 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-certs\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.515380 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.515352 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d25a72d-e345-4927-b69a-ecb298afa2f4-istiod-ca-cert\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.516715 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.516694 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-envoy\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.517020 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.517002 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-podinfo\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.524442 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.524418 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-token\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.525136 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.525095 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4fc9\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-kube-api-access-h4fc9\") pod \"e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.661282 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.661186 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:54.806481 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.806456 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj"] Jul 21 04:41:54.808208 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:41:54.808178 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d25a72d_e345_4927_b69a_ecb298afa2f4.slice/crio-d782e1028f05a1b3867206d465059c1360abebce08a40de2d4432c09c92dd737 WatchSource:0}: Error finding container d782e1028f05a1b3867206d465059c1360abebce08a40de2d4432c09c92dd737: Status 404 returned error can't find the container with id d782e1028f05a1b3867206d465059c1360abebce08a40de2d4432c09c92dd737 Jul 21 04:41:54.810431 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.810380 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:41:54.810525 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.810474 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:41:54.810567 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:54.810527 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:41:55.064796 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:55.064763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" event={"ID":"4d25a72d-e345-4927-b69a-ecb298afa2f4","Type":"ContainerStarted","Data":"5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c"} Jul 21 04:41:55.064971 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:55.064816 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" event={"ID":"4d25a72d-e345-4927-b69a-ecb298afa2f4","Type":"ContainerStarted","Data":"d782e1028f05a1b3867206d465059c1360abebce08a40de2d4432c09c92dd737"} Jul 21 04:41:55.086534 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:55.086481 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" podStartSLOduration=1.086464414 podStartE2EDuration="1.086464414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:41:55.084517784 +0000 UTC m=+1960.483850541" watchObservedRunningTime="2026-07-21 04:41:55.086464414 +0000 UTC m=+1960.485797172" Jul 21 04:41:55.661826 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:55.661787 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:55.666983 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:55.666955 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:56.069803 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:56.069771 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:41:56.070790 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:41:56.070767 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:42:02.077636 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:02.077595 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:42:10.159708 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:10.159614 2567 generic.go:356] "Generic (PLEG): container finished" podID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerID="5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c" exitCode=0 Jul 21 04:42:10.159708 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:10.159694 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" event={"ID":"4d25a72d-e345-4927-b69a-ecb298afa2f4","Type":"ContainerDied","Data":"5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c"} Jul 21 04:42:10.160188 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:10.160134 2567 scope.go:117] "RemoveContainer" containerID="5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c" Jul 21 04:42:10.161309 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:10.161277 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:42:10.161443 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:10.161371 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:42:10.161443 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:10.161408 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:42:11.070593 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:11.070553 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:42:11.166684 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:11.166648 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" event={"ID":"4d25a72d-e345-4927-b69a-ecb298afa2f4","Type":"ContainerStarted","Data":"11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591"} Jul 21 04:42:11.661749 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:11.661709 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:42:11.663050 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:11.663020 2567 patch_prober.go:28] interesting pod/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 21 04:42:11.663187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:11.663089 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 21 04:42:12.662589 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:12.662550 2567 patch_prober.go:28] interesting pod/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 21 04:42:12.662994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:12.662622 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 21 04:42:13.661718 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:13.661680 2567 patch_prober.go:28] interesting pod/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 21 04:42:13.661933 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:13.661742 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 21 04:42:14.661820 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:14.661784 2567 patch_prober.go:28] interesting pod/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 21 04:42:14.662227 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:14.661836 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 21 04:42:15.666397 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:15.666315 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:42:15.666793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:15.666675 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:42:15.667225 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:15.667204 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:42:20.168982 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:20.168949 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:42:41.879735 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:41.879697 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:42:59.774642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:42:59.774607 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:43:08.773036 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:43:08.772991 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:43:55.924631 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:43:55.924527 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj"] Jul 21 04:43:55.925226 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:43:55.924804 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" containerID="cri-o://11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591" gracePeriod=30 Jul 21 04:43:59.981563 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:43:59.981523 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:44:00.669329 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:00.669288 2567 patch_prober.go:28] interesting pod/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 04:44:00.669523 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:00.669365 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 04:44:01.182140 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.182115 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:44:01.254614 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.254526 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-token\") pod \"4d25a72d-e345-4927-b69a-ecb298afa2f4\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " Jul 21 04:44:01.254614 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.254566 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-data\") pod \"4d25a72d-e345-4927-b69a-ecb298afa2f4\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " Jul 21 04:44:01.254614 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.254611 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-podinfo\") pod \"4d25a72d-e345-4927-b69a-ecb298afa2f4\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " Jul 21 04:44:01.254899 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.254645 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-certs\") pod \"4d25a72d-e345-4927-b69a-ecb298afa2f4\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " Jul 21 04:44:01.254899 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.254690 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-socket\") pod \"4d25a72d-e345-4927-b69a-ecb298afa2f4\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " Jul 21 04:44:01.254899 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.254740 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-envoy\") pod \"4d25a72d-e345-4927-b69a-ecb298afa2f4\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " Jul 21 04:44:01.254899 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.254786 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4fc9\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-kube-api-access-h4fc9\") pod \"4d25a72d-e345-4927-b69a-ecb298afa2f4\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " Jul 21 04:44:01.254899 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.254817 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-credential-socket\") pod \"4d25a72d-e345-4927-b69a-ecb298afa2f4\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " Jul 21 04:44:01.254899 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.254871 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d25a72d-e345-4927-b69a-ecb298afa2f4-istiod-ca-cert\") pod \"4d25a72d-e345-4927-b69a-ecb298afa2f4\" (UID: \"4d25a72d-e345-4927-b69a-ecb298afa2f4\") " Jul 21 04:44:01.255240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.255116 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d25a72d-e345-4927-b69a-ecb298afa2f4" (UID: "4d25a72d-e345-4927-b69a-ecb298afa2f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:44:01.255240 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.255151 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d25a72d-e345-4927-b69a-ecb298afa2f4" (UID: "4d25a72d-e345-4927-b69a-ecb298afa2f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:44:01.255374 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.255309 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d25a72d-e345-4927-b69a-ecb298afa2f4" (UID: "4d25a72d-e345-4927-b69a-ecb298afa2f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:44:01.255529 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.255508 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d25a72d-e345-4927-b69a-ecb298afa2f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d25a72d-e345-4927-b69a-ecb298afa2f4" (UID: "4d25a72d-e345-4927-b69a-ecb298afa2f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:44:01.255634 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.255611 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d25a72d-e345-4927-b69a-ecb298afa2f4" (UID: "4d25a72d-e345-4927-b69a-ecb298afa2f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:44:01.256950 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.256927 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-kube-api-access-h4fc9" (OuterVolumeSpecName: "kube-api-access-h4fc9") pod "4d25a72d-e345-4927-b69a-ecb298afa2f4" (UID: "4d25a72d-e345-4927-b69a-ecb298afa2f4"). InnerVolumeSpecName "kube-api-access-h4fc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:44:01.257047 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.256927 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d25a72d-e345-4927-b69a-ecb298afa2f4" (UID: "4d25a72d-e345-4927-b69a-ecb298afa2f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 04:44:01.257386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.257354 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d25a72d-e345-4927-b69a-ecb298afa2f4" (UID: "4d25a72d-e345-4927-b69a-ecb298afa2f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:44:01.257386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.257363 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d25a72d-e345-4927-b69a-ecb298afa2f4" (UID: "4d25a72d-e345-4927-b69a-ecb298afa2f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:44:01.356439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.356404 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4fc9\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-kube-api-access-h4fc9\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:44:01.356439 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.356440 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-credential-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:44:01.356733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.356456 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d25a72d-e345-4927-b69a-ecb298afa2f4-istiod-ca-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:44:01.356733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.356469 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:44:01.356733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.356482 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-data\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:44:01.356733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.356496 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-podinfo\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:44:01.356733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.356508 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:44:01.356733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.356523 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-workload-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:44:01.356733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.356537 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d25a72d-e345-4927-b69a-ecb298afa2f4-istio-envoy\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:44:01.798276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.798241 2567 generic.go:356] "Generic (PLEG): container finished" podID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerID="11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591" exitCode=0 Jul 21 04:44:01.798518 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.798285 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" Jul 21 04:44:01.798518 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.798332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" event={"ID":"4d25a72d-e345-4927-b69a-ecb298afa2f4","Type":"ContainerDied","Data":"11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591"} Jul 21 04:44:01.798518 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.798413 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj" event={"ID":"4d25a72d-e345-4927-b69a-ecb298afa2f4","Type":"ContainerDied","Data":"d782e1028f05a1b3867206d465059c1360abebce08a40de2d4432c09c92dd737"} Jul 21 04:44:01.798518 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.798446 2567 scope.go:117] "RemoveContainer" containerID="11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591" Jul 21 04:44:01.810351 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.810314 2567 scope.go:117] "RemoveContainer" containerID="5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c" Jul 21 04:44:01.817761 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.817738 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj"] Jul 21 04:44:01.819354 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.819321 2567 scope.go:117] "RemoveContainer" containerID="11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591" Jul 21 04:44:01.819982 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:44:01.819959 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591\": container with ID starting with 11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591 not found: ID does not exist" containerID="11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591" Jul 21 04:44:01.820079 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.819993 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591"} err="failed to get container status \"11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591\": rpc error: code = NotFound desc = could not find container \"11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591\": container with ID starting with 11729746349aeb475b8a5eee6da495f2484cf7f3b4b84b926070ed0123e41591 not found: ID does not exist" Jul 21 04:44:01.820079 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.820011 2567 scope.go:117] "RemoveContainer" containerID="5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c" Jul 21 04:44:01.820299 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:44:01.820272 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c\": container with ID starting with 5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c not found: ID does not exist" containerID="5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c" Jul 21 04:44:01.820400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.820297 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c"} err="failed to get container status \"5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c\": rpc error: code = NotFound desc = could not find container \"5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c\": container with ID starting with 5b5821f94502a8142dbaf38596279130db6efd159469eff7390f7281aaf6247c not found: ID does not exist" Jul 21 04:44:01.824124 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:01.824103 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6ae222fa-openshift-default-58d8ff9b99-4fbrj"] Jul 21 04:44:03.267253 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:03.267223 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" path="/var/lib/kubelet/pods/4d25a72d-e345-4927-b69a-ecb298afa2f4/volumes" Jul 21 04:44:09.483220 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:09.483181 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:44:15.223733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:15.223699 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:44:15.229861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:15.229836 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:44:52.425958 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.425919 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2"] Jul 21 04:44:52.426482 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.426314 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" Jul 21 04:44:52.426482 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.426326 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" Jul 21 04:44:52.426482 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.426357 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" Jul 21 04:44:52.426482 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.426363 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" Jul 21 04:44:52.426482 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.426429 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="370dfb47-2616-49d4-938d-a557ca9e914e" containerName="istio-proxy" Jul 21 04:44:52.426482 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.426437 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" Jul 21 04:44:52.429492 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.429474 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.431865 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.431841 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-24ee3b-openshift-default-dockercfg-mr2tv\"" Jul 21 04:44:52.439674 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.439651 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2"] Jul 21 04:44:52.492035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.492001 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-istio-token\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.492035 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.492042 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglsm\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-kube-api-access-lglsm\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.492313 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.492118 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-certs\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.492313 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.492141 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-envoy\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.492313 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.492165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0648e6bd-9883-4558-930c-dec0979afeae-istiod-ca-cert\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.492313 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.492239 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-data\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.492313 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.492298 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-credential-socket\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.492580 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.492395 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-socket\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.492580 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.492421 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0648e6bd-9883-4558-930c-dec0979afeae-istio-podinfo\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.593407 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593333 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-certs\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.593407 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-envoy\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.593680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0648e6bd-9883-4558-930c-dec0979afeae-istiod-ca-cert\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.593680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593461 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-data\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.593680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-credential-socket\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.593680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-socket\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.593680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593584 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0648e6bd-9883-4558-930c-dec0979afeae-istio-podinfo\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.593680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593634 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-istio-token\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.593680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593669 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglsm\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-kube-api-access-lglsm\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.594043 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.593806 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-certs\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.594088 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.594059 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-credential-socket\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.594151 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.594062 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-data\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.594205 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.594157 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-socket\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.594258 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.594203 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0648e6bd-9883-4558-930c-dec0979afeae-istiod-ca-cert\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.595830 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.595807 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-envoy\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.596135 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.596113 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0648e6bd-9883-4558-930c-dec0979afeae-istio-podinfo\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.602559 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.602536 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-istio-token\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.602762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.602740 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglsm\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-kube-api-access-lglsm\") pod \"e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.740969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.740880 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:52.888249 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.888219 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2"] Jul 21 04:44:52.889512 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:44:52.889486 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0648e6bd_9883_4558_930c_dec0979afeae.slice/crio-347790131fa10133c7d7df02790849a80a2bd4cd5fdf86b7ab223503a0d7d8ef WatchSource:0}: Error finding container 347790131fa10133c7d7df02790849a80a2bd4cd5fdf86b7ab223503a0d7d8ef: Status 404 returned error can't find the container with id 347790131fa10133c7d7df02790849a80a2bd4cd5fdf86b7ab223503a0d7d8ef Jul 21 04:44:52.891686 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.891648 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:44:52.891763 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.891743 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:44:52.891803 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:52.891773 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:44:53.089486 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:53.089446 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" event={"ID":"0648e6bd-9883-4558-930c-dec0979afeae","Type":"ContainerStarted","Data":"02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff"} Jul 21 04:44:53.089677 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:53.089512 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" event={"ID":"0648e6bd-9883-4558-930c-dec0979afeae","Type":"ContainerStarted","Data":"347790131fa10133c7d7df02790849a80a2bd4cd5fdf86b7ab223503a0d7d8ef"} Jul 21 04:44:53.111448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:53.111387 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" podStartSLOduration=1.111369881 podStartE2EDuration="1.111369881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:44:53.110143186 +0000 UTC m=+2138.509475956" watchObservedRunningTime="2026-07-21 04:44:53.111369881 +0000 UTC m=+2138.510702645" Jul 21 04:44:53.741238 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:53.741202 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:53.746730 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:53.746702 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:54.098603 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:54.098565 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:44:54.099529 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:44:54.099507 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:45:01.377034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:01.377001 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:45:09.138175 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:09.137955 2567 patch_prober.go:28] interesting pod/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:53458->10.132.0.66:15021: read: connection reset by peer" start-of-body= Jul 21 04:45:09.138175 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:09.138040 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:53458->10.132.0.66:15021: read: connection reset by peer" Jul 21 04:45:09.197289 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:09.197262 2567 generic.go:356] "Generic (PLEG): container finished" podID="0648e6bd-9883-4558-930c-dec0979afeae" containerID="02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff" exitCode=0 Jul 21 04:45:09.197441 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:09.197354 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" event={"ID":"0648e6bd-9883-4558-930c-dec0979afeae","Type":"ContainerDied","Data":"02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff"} Jul 21 04:45:09.197752 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:09.197738 2567 scope.go:117] "RemoveContainer" containerID="02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff" Jul 21 04:45:09.198824 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:09.198785 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:45:09.198894 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:09.198870 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:45:09.198931 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:09.198902 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:45:10.204397 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:10.204360 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" event={"ID":"0648e6bd-9883-4558-930c-dec0979afeae","Type":"ContainerStarted","Data":"568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd"} Jul 21 04:45:10.741940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:10.741904 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:45:10.742229 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:10.742198 2567 patch_prober.go:28] interesting pod/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 21 04:45:10.742434 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:10.742404 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 21 04:45:11.741985 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:11.741937 2567 patch_prober.go:28] interesting pod/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 21 04:45:11.742521 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:11.742021 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 21 04:45:12.742230 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:12.742186 2567 patch_prober.go:28] interesting pod/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 21 04:45:12.742894 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:12.742859 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 21 04:45:13.742254 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:13.742217 2567 patch_prober.go:28] interesting pod/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 21 04:45:13.742860 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:13.742827 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 21 04:45:14.744817 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:14.744784 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:45:14.745209 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:14.745032 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:45:14.745622 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:14.745603 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:45:16.779361 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:16.779307 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:45:19.177037 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.177002 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr"] Jul 21 04:45:19.177758 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.177735 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d25a72d-e345-4927-b69a-ecb298afa2f4" containerName="istio-proxy" Jul 21 04:45:19.180253 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.180234 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.183943 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.183787 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-24ee3b\"/\"default-dockercfg-h8gsn\"" Jul 21 04:45:19.183943 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.183801 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-24ee3b\"/\"test-model-24ee3b-kserve-self-signed-certs\"" Jul 21 04:45:19.183943 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.183812 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-24ee3b\"/\"openshift-service-ca.crt\"" Jul 21 04:45:19.183943 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.183827 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-24ee3b\"/\"kube-root-ca.crt\"" Jul 21 04:45:19.189796 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.189194 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr"] Jul 21 04:45:19.349624 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.349591 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-home\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.349808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.349648 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkcjg\" (UniqueName: \"kubernetes.io/projected/67fbe5de-a66f-474d-9ed8-67b21db51503-kube-api-access-hkcjg\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.349808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.349712 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-dshm\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.349808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.349795 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67fbe5de-a66f-474d-9ed8-67b21db51503-tls-certs\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.349940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.349819 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-model-cache\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.349940 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.349888 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-tmp-dir\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.451642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.451556 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkcjg\" (UniqueName: \"kubernetes.io/projected/67fbe5de-a66f-474d-9ed8-67b21db51503-kube-api-access-hkcjg\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.451642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.451594 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-dshm\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.451642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.451638 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67fbe5de-a66f-474d-9ed8-67b21db51503-tls-certs\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.451954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.451659 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-model-cache\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.451954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.451683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-tmp-dir\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.451954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.451797 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-home\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.452113 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.452029 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-tmp-dir\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.452113 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.452090 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-model-cache\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.452189 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.452173 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-home\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.453892 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.453873 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-dshm\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.454200 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.454183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67fbe5de-a66f-474d-9ed8-67b21db51503-tls-certs\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.459875 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.459850 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkcjg\" (UniqueName: \"kubernetes.io/projected/67fbe5de-a66f-474d-9ed8-67b21db51503-kube-api-access-hkcjg\") pod \"test-model-24ee3b-kserve-68dd5b6f9f-4zghr\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.492989 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.492968 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:19.632363 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:19.632316 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr"] Jul 21 04:45:19.633700 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:45:19.633673 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67fbe5de_a66f_474d_9ed8_67b21db51503.slice/crio-87302954e6b48b1eabfcbbdeb48895583689a836ef5f2208d2132cf2b15dce09 WatchSource:0}: Error finding container 87302954e6b48b1eabfcbbdeb48895583689a836ef5f2208d2132cf2b15dce09: Status 404 returned error can't find the container with id 87302954e6b48b1eabfcbbdeb48895583689a836ef5f2208d2132cf2b15dce09 Jul 21 04:45:20.261844 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:20.261802 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" event={"ID":"67fbe5de-a66f-474d-9ed8-67b21db51503","Type":"ContainerStarted","Data":"042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163"} Jul 21 04:45:20.261844 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:20.261853 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" event={"ID":"67fbe5de-a66f-474d-9ed8-67b21db51503","Type":"ContainerStarted","Data":"87302954e6b48b1eabfcbbdeb48895583689a836ef5f2208d2132cf2b15dce09"} Jul 21 04:45:20.279008 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:20.278954 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" podStartSLOduration=1.278939207 podStartE2EDuration="1.278939207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:45:20.277925456 +0000 UTC m=+2165.677258213" watchObservedRunningTime="2026-07-21 04:45:20.278939207 +0000 UTC m=+2165.678271966" Jul 21 04:45:24.482255 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:24.482226 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:45:29.493898 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:29.493859 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:29.494291 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:29.493911 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:29.506492 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:29.506466 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:30.327852 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:30.327820 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:45:35.570643 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:35.570608 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:45:38.476925 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.476894 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql"] Jul 21 04:45:38.479891 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.479872 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.482246 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.482226 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-97b818\"/\"kube-root-ca.crt\"" Jul 21 04:45:38.483275 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.483251 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-97b818\"/\"default-dockercfg-gw4ln\"" Jul 21 04:45:38.483403 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.483281 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-97b818\"/\"openshift-service-ca.crt\"" Jul 21 04:45:38.483403 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.483282 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-97b818\"/\"test-model-97b818-kserve-self-signed-certs\"" Jul 21 04:45:38.488818 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.488799 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql"] Jul 21 04:45:38.509728 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.509699 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-dshm\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.509862 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.509778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-tmp-dir\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.509862 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.509812 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-model-cache\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.509993 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.509870 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-home\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.509993 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.509904 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blb7p\" (UniqueName: \"kubernetes.io/projected/93324fa2-c77f-4ace-b01c-278021305879-kube-api-access-blb7p\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.509993 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.509931 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93324fa2-c77f-4ace-b01c-278021305879-tls-certs\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.610662 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.610624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-home\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.610843 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.610670 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blb7p\" (UniqueName: \"kubernetes.io/projected/93324fa2-c77f-4ace-b01c-278021305879-kube-api-access-blb7p\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.610843 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.610698 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93324fa2-c77f-4ace-b01c-278021305879-tls-certs\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.610843 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.610730 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-dshm\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.610843 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.610803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-tmp-dir\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.610843 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.610835 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-model-cache\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.611127 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.611064 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-home\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.611215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.611193 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-model-cache\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.611326 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.611307 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-tmp-dir\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.613140 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.613119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-dshm\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.613330 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.613312 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93324fa2-c77f-4ace-b01c-278021305879-tls-certs\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.621045 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.621008 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blb7p\" (UniqueName: \"kubernetes.io/projected/93324fa2-c77f-4ace-b01c-278021305879-kube-api-access-blb7p\") pod \"test-model-97b818-kserve-864dc4dff9-zfmql\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.790526 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.790483 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:38.928693 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:38.928665 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql"] Jul 21 04:45:38.930226 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:45:38.930193 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93324fa2_c77f_4ace_b01c_278021305879.slice/crio-a88ae8b91dff92a6addc9eabc12dc7ca628cb2755fa4d9a6387b349d9f3936a6 WatchSource:0}: Error finding container a88ae8b91dff92a6addc9eabc12dc7ca628cb2755fa4d9a6387b349d9f3936a6: Status 404 returned error can't find the container with id a88ae8b91dff92a6addc9eabc12dc7ca628cb2755fa4d9a6387b349d9f3936a6 Jul 21 04:45:39.368249 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:39.368217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" event={"ID":"93324fa2-c77f-4ace-b01c-278021305879","Type":"ContainerStarted","Data":"bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c"} Jul 21 04:45:39.368488 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:39.368265 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" event={"ID":"93324fa2-c77f-4ace-b01c-278021305879","Type":"ContainerStarted","Data":"a88ae8b91dff92a6addc9eabc12dc7ca628cb2755fa4d9a6387b349d9f3936a6"} Jul 21 04:45:39.384316 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:39.384261 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" podStartSLOduration=1.384246288 podStartE2EDuration="1.384246288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:45:39.383357699 +0000 UTC m=+2184.782690450" watchObservedRunningTime="2026-07-21 04:45:39.384246288 +0000 UTC m=+2184.783579045" Jul 21 04:45:46.679086 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:46.679046 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:45:48.790645 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:48.790607 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:48.791046 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:48.790658 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:48.802907 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:48.802881 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:45:49.439770 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:45:49.439727 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:46:22.135189 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:22.135150 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr"] Jul 21 04:46:23.075246 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:23.075201 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql"] Jul 21 04:46:27.172404 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:27.172362 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:46:37.156688 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.156643 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" podUID="67fbe5de-a66f-474d-9ed8-67b21db51503" containerName="main" containerID="cri-o://042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163" gracePeriod=45 Jul 21 04:46:37.419004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.418928 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:46:37.566654 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.566612 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67fbe5de-a66f-474d-9ed8-67b21db51503-tls-certs\") pod \"67fbe5de-a66f-474d-9ed8-67b21db51503\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " Jul 21 04:46:37.566861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.566665 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-model-cache\") pod \"67fbe5de-a66f-474d-9ed8-67b21db51503\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " Jul 21 04:46:37.566861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.566789 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-tmp-dir\") pod \"67fbe5de-a66f-474d-9ed8-67b21db51503\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " Jul 21 04:46:37.566861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.566823 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkcjg\" (UniqueName: \"kubernetes.io/projected/67fbe5de-a66f-474d-9ed8-67b21db51503-kube-api-access-hkcjg\") pod \"67fbe5de-a66f-474d-9ed8-67b21db51503\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " Jul 21 04:46:37.566861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.566848 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-dshm\") pod \"67fbe5de-a66f-474d-9ed8-67b21db51503\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " Jul 21 04:46:37.567086 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.566880 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-home\") pod \"67fbe5de-a66f-474d-9ed8-67b21db51503\" (UID: \"67fbe5de-a66f-474d-9ed8-67b21db51503\") " Jul 21 04:46:37.567086 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.566992 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-model-cache" (OuterVolumeSpecName: "model-cache") pod "67fbe5de-a66f-474d-9ed8-67b21db51503" (UID: "67fbe5de-a66f-474d-9ed8-67b21db51503"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:37.567171 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.567078 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67fbe5de-a66f-474d-9ed8-67b21db51503" (UID: "67fbe5de-a66f-474d-9ed8-67b21db51503"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:37.567234 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.567204 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-home" (OuterVolumeSpecName: "home") pod "67fbe5de-a66f-474d-9ed8-67b21db51503" (UID: "67fbe5de-a66f-474d-9ed8-67b21db51503"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:37.567534 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.567515 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-tmp-dir\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:37.567534 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.567535 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-home\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:37.567711 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.567548 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-model-cache\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:37.569034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.569011 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67fbe5de-a66f-474d-9ed8-67b21db51503-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67fbe5de-a66f-474d-9ed8-67b21db51503" (UID: "67fbe5de-a66f-474d-9ed8-67b21db51503"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:46:37.569114 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.569010 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-dshm" (OuterVolumeSpecName: "dshm") pod "67fbe5de-a66f-474d-9ed8-67b21db51503" (UID: "67fbe5de-a66f-474d-9ed8-67b21db51503"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:37.569114 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.569057 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fbe5de-a66f-474d-9ed8-67b21db51503-kube-api-access-hkcjg" (OuterVolumeSpecName: "kube-api-access-hkcjg") pod "67fbe5de-a66f-474d-9ed8-67b21db51503" (UID: "67fbe5de-a66f-474d-9ed8-67b21db51503"). InnerVolumeSpecName "kube-api-access-hkcjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:46:37.668894 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.668857 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkcjg\" (UniqueName: \"kubernetes.io/projected/67fbe5de-a66f-474d-9ed8-67b21db51503-kube-api-access-hkcjg\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:37.668894 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.668890 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67fbe5de-a66f-474d-9ed8-67b21db51503-dshm\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:37.668894 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.668903 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67fbe5de-a66f-474d-9ed8-67b21db51503-tls-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:37.691973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.691879 2567 generic.go:356] "Generic (PLEG): container finished" podID="67fbe5de-a66f-474d-9ed8-67b21db51503" containerID="042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163" exitCode=0 Jul 21 04:46:37.691973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.691924 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" Jul 21 04:46:37.692159 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.691964 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" event={"ID":"67fbe5de-a66f-474d-9ed8-67b21db51503","Type":"ContainerDied","Data":"042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163"} Jul 21 04:46:37.692159 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.692016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr" event={"ID":"67fbe5de-a66f-474d-9ed8-67b21db51503","Type":"ContainerDied","Data":"87302954e6b48b1eabfcbbdeb48895583689a836ef5f2208d2132cf2b15dce09"} Jul 21 04:46:37.692159 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.692035 2567 scope.go:117] "RemoveContainer" containerID="042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163" Jul 21 04:46:37.702440 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.702417 2567 scope.go:117] "RemoveContainer" containerID="042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163" Jul 21 04:46:37.702748 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:46:37.702725 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163\": container with ID starting with 042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163 not found: ID does not exist" containerID="042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163" Jul 21 04:46:37.702809 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.702758 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163"} err="failed to get container status \"042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163\": rpc error: code = NotFound desc = could not find container \"042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163\": container with ID starting with 042badaba79ace176d4158931f82a4d4a6ca3648f130c6e979cb7f77bb5bb163 not found: ID does not exist" Jul 21 04:46:37.714867 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.714822 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr"] Jul 21 04:46:37.717279 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:37.717258 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-24ee3b/test-model-24ee3b-kserve-68dd5b6f9f-4zghr"] Jul 21 04:46:38.096501 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.096441 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" podUID="93324fa2-c77f-4ace-b01c-278021305879" containerName="main" containerID="cri-o://bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c" gracePeriod=45 Jul 21 04:46:38.361475 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.361410 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:46:38.477714 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.477671 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93324fa2-c77f-4ace-b01c-278021305879-tls-certs\") pod \"93324fa2-c77f-4ace-b01c-278021305879\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " Jul 21 04:46:38.477714 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.477716 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-model-cache\") pod \"93324fa2-c77f-4ace-b01c-278021305879\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " Jul 21 04:46:38.477973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.477758 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blb7p\" (UniqueName: \"kubernetes.io/projected/93324fa2-c77f-4ace-b01c-278021305879-kube-api-access-blb7p\") pod \"93324fa2-c77f-4ace-b01c-278021305879\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " Jul 21 04:46:38.477973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.477813 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-tmp-dir\") pod \"93324fa2-c77f-4ace-b01c-278021305879\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " Jul 21 04:46:38.477973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.477844 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-home\") pod \"93324fa2-c77f-4ace-b01c-278021305879\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " Jul 21 04:46:38.477973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.477884 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-dshm\") pod \"93324fa2-c77f-4ace-b01c-278021305879\" (UID: \"93324fa2-c77f-4ace-b01c-278021305879\") " Jul 21 04:46:38.478187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.478068 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93324fa2-c77f-4ace-b01c-278021305879" (UID: "93324fa2-c77f-4ace-b01c-278021305879"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:38.478187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.478155 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-home" (OuterVolumeSpecName: "home") pod "93324fa2-c77f-4ace-b01c-278021305879" (UID: "93324fa2-c77f-4ace-b01c-278021305879"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:38.478187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.478078 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-model-cache" (OuterVolumeSpecName: "model-cache") pod "93324fa2-c77f-4ace-b01c-278021305879" (UID: "93324fa2-c77f-4ace-b01c-278021305879"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:38.478311 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.478248 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-model-cache\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:38.478311 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.478271 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-tmp-dir\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:38.478311 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.478283 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-home\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:38.480161 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.480136 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93324fa2-c77f-4ace-b01c-278021305879-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93324fa2-c77f-4ace-b01c-278021305879" (UID: "93324fa2-c77f-4ace-b01c-278021305879"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:46:38.480161 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.480148 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-dshm" (OuterVolumeSpecName: "dshm") pod "93324fa2-c77f-4ace-b01c-278021305879" (UID: "93324fa2-c77f-4ace-b01c-278021305879"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:38.480352 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.480200 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93324fa2-c77f-4ace-b01c-278021305879-kube-api-access-blb7p" (OuterVolumeSpecName: "kube-api-access-blb7p") pod "93324fa2-c77f-4ace-b01c-278021305879" (UID: "93324fa2-c77f-4ace-b01c-278021305879"). InnerVolumeSpecName "kube-api-access-blb7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:46:38.579635 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.579589 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blb7p\" (UniqueName: \"kubernetes.io/projected/93324fa2-c77f-4ace-b01c-278021305879-kube-api-access-blb7p\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:38.579635 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.579628 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93324fa2-c77f-4ace-b01c-278021305879-dshm\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:38.579635 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.579637 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93324fa2-c77f-4ace-b01c-278021305879-tls-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:38.700071 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.699975 2567 generic.go:356] "Generic (PLEG): container finished" podID="93324fa2-c77f-4ace-b01c-278021305879" containerID="bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c" exitCode=0 Jul 21 04:46:38.700071 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.700023 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" Jul 21 04:46:38.700071 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.700055 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" event={"ID":"93324fa2-c77f-4ace-b01c-278021305879","Type":"ContainerDied","Data":"bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c"} Jul 21 04:46:38.700349 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.700100 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql" event={"ID":"93324fa2-c77f-4ace-b01c-278021305879","Type":"ContainerDied","Data":"a88ae8b91dff92a6addc9eabc12dc7ca628cb2755fa4d9a6387b349d9f3936a6"} Jul 21 04:46:38.700349 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.700119 2567 scope.go:117] "RemoveContainer" containerID="bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c" Jul 21 04:46:38.711191 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.711162 2567 scope.go:117] "RemoveContainer" containerID="bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c" Jul 21 04:46:38.711667 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:46:38.711633 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c\": container with ID starting with bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c not found: ID does not exist" containerID="bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c" Jul 21 04:46:38.711815 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.711680 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c"} err="failed to get container status \"bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c\": rpc error: code = NotFound desc = could not find container \"bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c\": container with ID starting with bf43df7c76c6e5bbbb09940032271efb4b1d75481f679d428c98ca9500577d9c not found: ID does not exist" Jul 21 04:46:38.730869 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.730832 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql"] Jul 21 04:46:38.734861 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:38.734831 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-97b818/test-model-97b818-kserve-864dc4dff9-zfmql"] Jul 21 04:46:39.267502 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:39.267465 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67fbe5de-a66f-474d-9ed8-67b21db51503" path="/var/lib/kubelet/pods/67fbe5de-a66f-474d-9ed8-67b21db51503/volumes" Jul 21 04:46:39.267881 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:39.267867 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93324fa2-c77f-4ace-b01c-278021305879" path="/var/lib/kubelet/pods/93324fa2-c77f-4ace-b01c-278021305879/volumes" Jul 21 04:46:43.077070 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:43.077027 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:46:45.092774 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:45.092740 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2"] Jul 21 04:46:45.093265 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:45.093008 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" containerID="cri-o://568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd" gracePeriod=30 Jul 21 04:46:50.351810 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.351781 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:46:50.492282 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492187 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-certs\") pod \"0648e6bd-9883-4558-930c-dec0979afeae\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " Jul 21 04:46:50.492282 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492251 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-istio-token\") pod \"0648e6bd-9883-4558-930c-dec0979afeae\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " Jul 21 04:46:50.492282 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492280 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-envoy\") pod \"0648e6bd-9883-4558-930c-dec0979afeae\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " Jul 21 04:46:50.492573 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492313 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0648e6bd-9883-4558-930c-dec0979afeae-istiod-ca-cert\") pod \"0648e6bd-9883-4558-930c-dec0979afeae\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " Jul 21 04:46:50.492573 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492485 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-socket\") pod \"0648e6bd-9883-4558-930c-dec0979afeae\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " Jul 21 04:46:50.492768 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492572 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0648e6bd-9883-4558-930c-dec0979afeae" (UID: "0648e6bd-9883-4558-930c-dec0979afeae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:50.492768 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492577 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-credential-socket\") pod \"0648e6bd-9883-4558-930c-dec0979afeae\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " Jul 21 04:46:50.492768 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492691 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0648e6bd-9883-4558-930c-dec0979afeae-istio-podinfo\") pod \"0648e6bd-9883-4558-930c-dec0979afeae\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " Jul 21 04:46:50.492768 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492730 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lglsm\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-kube-api-access-lglsm\") pod \"0648e6bd-9883-4558-930c-dec0979afeae\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " Jul 21 04:46:50.492968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492781 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-data\") pod \"0648e6bd-9883-4558-930c-dec0979afeae\" (UID: \"0648e6bd-9883-4558-930c-dec0979afeae\") " Jul 21 04:46:50.492968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492786 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0648e6bd-9883-4558-930c-dec0979afeae" (UID: "0648e6bd-9883-4558-930c-dec0979afeae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:50.492968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492800 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0648e6bd-9883-4558-930c-dec0979afeae" (UID: "0648e6bd-9883-4558-930c-dec0979afeae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:50.492968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.492792 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0648e6bd-9883-4558-930c-dec0979afeae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0648e6bd-9883-4558-930c-dec0979afeae" (UID: "0648e6bd-9883-4558-930c-dec0979afeae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:46:50.493175 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.493089 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-credential-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:50.493175 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.493107 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:50.493175 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.493117 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0648e6bd-9883-4558-930c-dec0979afeae-istiod-ca-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:50.493175 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.493125 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-workload-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:50.493740 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.493714 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-data" (OuterVolumeSpecName: "istio-data") pod "0648e6bd-9883-4558-930c-dec0979afeae" (UID: "0648e6bd-9883-4558-930c-dec0979afeae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:50.494724 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.494693 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0648e6bd-9883-4558-930c-dec0979afeae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0648e6bd-9883-4558-930c-dec0979afeae" (UID: "0648e6bd-9883-4558-930c-dec0979afeae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 04:46:50.494813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.494732 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0648e6bd-9883-4558-930c-dec0979afeae" (UID: "0648e6bd-9883-4558-930c-dec0979afeae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:46:50.494926 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.494903 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-istio-token" (OuterVolumeSpecName: "istio-token") pod "0648e6bd-9883-4558-930c-dec0979afeae" (UID: "0648e6bd-9883-4558-930c-dec0979afeae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:46:50.495005 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.494987 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-kube-api-access-lglsm" (OuterVolumeSpecName: "kube-api-access-lglsm") pod "0648e6bd-9883-4558-930c-dec0979afeae" (UID: "0648e6bd-9883-4558-930c-dec0979afeae"). InnerVolumeSpecName "kube-api-access-lglsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:46:50.593788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.593751 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:50.593788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.593780 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-envoy\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:50.593788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.593789 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0648e6bd-9883-4558-930c-dec0979afeae-istio-podinfo\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:50.593788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.593798 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lglsm\" (UniqueName: \"kubernetes.io/projected/0648e6bd-9883-4558-930c-dec0979afeae-kube-api-access-lglsm\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:50.594076 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.593807 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0648e6bd-9883-4558-930c-dec0979afeae-istio-data\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:46:50.772261 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.772232 2567 generic.go:356] "Generic (PLEG): container finished" podID="0648e6bd-9883-4558-930c-dec0979afeae" containerID="568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd" exitCode=0 Jul 21 04:46:50.772491 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.772282 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" Jul 21 04:46:50.772491 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.772312 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" event={"ID":"0648e6bd-9883-4558-930c-dec0979afeae","Type":"ContainerDied","Data":"568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd"} Jul 21 04:46:50.772491 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.772394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2" event={"ID":"0648e6bd-9883-4558-930c-dec0979afeae","Type":"ContainerDied","Data":"347790131fa10133c7d7df02790849a80a2bd4cd5fdf86b7ab223503a0d7d8ef"} Jul 21 04:46:50.772491 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.772416 2567 scope.go:117] "RemoveContainer" containerID="568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd" Jul 21 04:46:50.785724 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.785699 2567 scope.go:117] "RemoveContainer" containerID="02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff" Jul 21 04:46:50.794400 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.794381 2567 scope.go:117] "RemoveContainer" containerID="568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd" Jul 21 04:46:50.794723 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:46:50.794702 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd\": container with ID starting with 568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd not found: ID does not exist" containerID="568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd" Jul 21 04:46:50.794812 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.794731 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd"} err="failed to get container status \"568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd\": rpc error: code = NotFound desc = could not find container \"568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd\": container with ID starting with 568b23bf2448ce333997401c05cb2979c3f7b2e67ae1b2e3440d3242c44536dd not found: ID does not exist" Jul 21 04:46:50.794812 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.794750 2567 scope.go:117] "RemoveContainer" containerID="02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff" Jul 21 04:46:50.795156 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:46:50.795126 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff\": container with ID starting with 02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff not found: ID does not exist" containerID="02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff" Jul 21 04:46:50.795271 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.795166 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff"} err="failed to get container status \"02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff\": rpc error: code = NotFound desc = could not find container \"02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff\": container with ID starting with 02932af9aeb8389d8d5ad863fe4a07d6f7b11c688f26e4c0e1d16f8186357eff not found: ID does not exist" Jul 21 04:46:50.798256 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.798237 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2"] Jul 21 04:46:50.802600 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:50.802579 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-24ee3b-openshift-default-5d868dc55-ztfb2"] Jul 21 04:46:51.266442 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:51.266411 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0648e6bd-9883-4558-930c-dec0979afeae" path="/var/lib/kubelet/pods/0648e6bd-9883-4558-930c-dec0979afeae/volumes" Jul 21 04:46:58.976615 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:46:58.976572 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:47:07.969584 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.969504 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln"] Jul 21 04:47:07.969991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.969929 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" Jul 21 04:47:07.969991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.969941 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" Jul 21 04:47:07.969991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.969951 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93324fa2-c77f-4ace-b01c-278021305879" containerName="main" Jul 21 04:47:07.969991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.969957 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="93324fa2-c77f-4ace-b01c-278021305879" containerName="main" Jul 21 04:47:07.969991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.969967 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" Jul 21 04:47:07.969991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.969974 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" Jul 21 04:47:07.969991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.969983 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fbe5de-a66f-474d-9ed8-67b21db51503" containerName="main" Jul 21 04:47:07.969991 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.969988 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fbe5de-a66f-474d-9ed8-67b21db51503" containerName="main" Jul 21 04:47:07.970241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.970046 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" Jul 21 04:47:07.970241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.970055 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0648e6bd-9883-4558-930c-dec0979afeae" containerName="istio-proxy" Jul 21 04:47:07.970241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.970062 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="67fbe5de-a66f-474d-9ed8-67b21db51503" containerName="main" Jul 21 04:47:07.970241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.970074 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="93324fa2-c77f-4ace-b01c-278021305879" containerName="main" Jul 21 04:47:07.974756 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.974737 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:07.977183 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.977164 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f63636-openshift-default-dockercfg-pw48v\"" Jul 21 04:47:07.988513 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:07.988490 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln"] Jul 21 04:47:08.042525 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.042486 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-istio-token\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.042691 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.042533 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-socket\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.042691 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.042550 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-credential-socket\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.042691 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.042600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-envoy\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.042691 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.042623 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-certs\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.042691 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.042643 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-data\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.042691 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.042677 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnvz\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-kube-api-access-5hnvz\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.042894 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.042698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d71200-f415-498e-89ba-d10019767ebf-istiod-ca-cert\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.042894 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.042715 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d71200-f415-498e-89ba-d10019767ebf-istio-podinfo\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.143959 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnvz\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-kube-api-access-5hnvz\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144004 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144010 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d71200-f415-498e-89ba-d10019767ebf-istiod-ca-cert\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144102 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d71200-f415-498e-89ba-d10019767ebf-istio-podinfo\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144145 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-istio-token\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144176 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-socket\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144201 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-credential-socket\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144246 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-envoy\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144542 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144286 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-certs\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144542 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-data\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144725 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144670 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-socket\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144876 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144745 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-credential-socket\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.144876 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144807 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-data\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.145001 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144893 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-certs\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.145001 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.144982 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d71200-f415-498e-89ba-d10019767ebf-istiod-ca-cert\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.146618 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.146595 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-envoy\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.146819 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.146804 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d71200-f415-498e-89ba-d10019767ebf-istio-podinfo\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.152747 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.152725 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnvz\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-kube-api-access-5hnvz\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.152747 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.152735 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-istio-token\") pod \"e2e-api-a-f63636-openshift-default-cd4c64655-tcmln\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.285674 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.285640 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:08.434466 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.434440 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln"] Jul 21 04:47:08.435757 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:47:08.435725 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d71200_f415_498e_89ba_d10019767ebf.slice/crio-09bc2503c7187dab6aa11c7beadc15eddffa6cfb20311aab702dc5e02e929f76 WatchSource:0}: Error finding container 09bc2503c7187dab6aa11c7beadc15eddffa6cfb20311aab702dc5e02e929f76: Status 404 returned error can't find the container with id 09bc2503c7187dab6aa11c7beadc15eddffa6cfb20311aab702dc5e02e929f76 Jul 21 04:47:08.437973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.437934 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:47:08.438046 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.438027 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:47:08.438092 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.438074 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:47:08.876994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.876956 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" event={"ID":"b0d71200-f415-498e-89ba-d10019767ebf","Type":"ContainerStarted","Data":"df7e90602aac1395ea59017bd0262fd679c631d55d6959f28c17351d4647a8e5"} Jul 21 04:47:08.877178 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.877011 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" event={"ID":"b0d71200-f415-498e-89ba-d10019767ebf","Type":"ContainerStarted","Data":"09bc2503c7187dab6aa11c7beadc15eddffa6cfb20311aab702dc5e02e929f76"} Jul 21 04:47:08.899237 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:08.899186 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" podStartSLOduration=1.899170075 podStartE2EDuration="1.899170075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:47:08.896645981 +0000 UTC m=+2274.295978740" watchObservedRunningTime="2026-07-21 04:47:08.899170075 +0000 UTC m=+2274.298502833" Jul 21 04:47:09.286230 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:09.286201 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:09.291023 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:09.290997 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:09.887891 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:09.887851 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:09.890163 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:09.890134 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:13.883183 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:13.883142 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:47:21.969476 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:21.969394 2567 generic.go:356] "Generic (PLEG): container finished" podID="b0d71200-f415-498e-89ba-d10019767ebf" containerID="df7e90602aac1395ea59017bd0262fd679c631d55d6959f28c17351d4647a8e5" exitCode=0 Jul 21 04:47:21.969476 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:21.969432 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" event={"ID":"b0d71200-f415-498e-89ba-d10019767ebf","Type":"ContainerDied","Data":"df7e90602aac1395ea59017bd0262fd679c631d55d6959f28c17351d4647a8e5"} Jul 21 04:47:21.969926 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:21.969823 2567 scope.go:117] "RemoveContainer" containerID="df7e90602aac1395ea59017bd0262fd679c631d55d6959f28c17351d4647a8e5" Jul 21 04:47:21.970884 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:21.970846 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:47:21.970994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:21.970919 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:47:21.970994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:21.970961 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:47:22.977207 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:22.977167 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" event={"ID":"b0d71200-f415-498e-89ba-d10019767ebf","Type":"ContainerStarted","Data":"811782f65acc22be105b113d369ec07403d8a07905bbe43f890b3dbbb9a489aa"} Jul 21 04:47:23.286417 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:23.286386 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:23.287692 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:23.287665 2567 patch_prober.go:28] interesting pod/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 21 04:47:23.287811 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:23.287729 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 21 04:47:24.286964 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:24.286924 2567 patch_prober.go:28] interesting pod/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 21 04:47:24.287385 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:24.286998 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 21 04:47:24.887878 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:24.887839 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:25.286381 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:25.286327 2567 patch_prober.go:28] interesting pod/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 21 04:47:25.286582 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:25.286403 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 21 04:47:26.286219 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:26.286180 2567 patch_prober.go:28] interesting pod/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 21 04:47:26.286657 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:26.286231 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 21 04:47:27.290010 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:27.289978 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:27.291142 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:27.291122 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:47:32.475489 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:32.475457 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:47:50.872794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:50.872758 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:47:58.036594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:58.036559 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln"] Jul 21 04:47:58.037064 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:47:58.036773 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" containerID="cri-o://811782f65acc22be105b113d369ec07403d8a07905bbe43f890b3dbbb9a489aa" gracePeriod=30 Jul 21 04:48:03.214219 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.214189 2567 generic.go:356] "Generic (PLEG): container finished" podID="b0d71200-f415-498e-89ba-d10019767ebf" containerID="811782f65acc22be105b113d369ec07403d8a07905bbe43f890b3dbbb9a489aa" exitCode=0 Jul 21 04:48:03.214679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.214274 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" event={"ID":"b0d71200-f415-498e-89ba-d10019767ebf","Type":"ContainerDied","Data":"811782f65acc22be105b113d369ec07403d8a07905bbe43f890b3dbbb9a489aa"} Jul 21 04:48:03.214679 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.214425 2567 scope.go:117] "RemoveContainer" containerID="df7e90602aac1395ea59017bd0262fd679c631d55d6959f28c17351d4647a8e5" Jul 21 04:48:03.299714 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.299689 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:48:03.402733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402635 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-socket\") pod \"b0d71200-f415-498e-89ba-d10019767ebf\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " Jul 21 04:48:03.402733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402687 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d71200-f415-498e-89ba-d10019767ebf-istio-podinfo\") pod \"b0d71200-f415-498e-89ba-d10019767ebf\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " Jul 21 04:48:03.402973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402746 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-istio-token\") pod \"b0d71200-f415-498e-89ba-d10019767ebf\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " Jul 21 04:48:03.402973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402804 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-certs\") pod \"b0d71200-f415-498e-89ba-d10019767ebf\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " Jul 21 04:48:03.402973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402838 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-data\") pod \"b0d71200-f415-498e-89ba-d10019767ebf\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " Jul 21 04:48:03.402973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402864 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-envoy\") pod \"b0d71200-f415-498e-89ba-d10019767ebf\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " Jul 21 04:48:03.402973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402884 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d71200-f415-498e-89ba-d10019767ebf-istiod-ca-cert\") pod \"b0d71200-f415-498e-89ba-d10019767ebf\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " Jul 21 04:48:03.402973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402919 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hnvz\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-kube-api-access-5hnvz\") pod \"b0d71200-f415-498e-89ba-d10019767ebf\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " Jul 21 04:48:03.402973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402953 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-credential-socket\") pod \"b0d71200-f415-498e-89ba-d10019767ebf\" (UID: \"b0d71200-f415-498e-89ba-d10019767ebf\") " Jul 21 04:48:03.402973 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.402952 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b0d71200-f415-498e-89ba-d10019767ebf" (UID: "b0d71200-f415-498e-89ba-d10019767ebf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:48:03.403406 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.403215 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b0d71200-f415-498e-89ba-d10019767ebf" (UID: "b0d71200-f415-498e-89ba-d10019767ebf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:48:03.403406 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.403308 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:48:03.403406 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.403325 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-workload-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:48:03.403406 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.403330 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d71200-f415-498e-89ba-d10019767ebf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b0d71200-f415-498e-89ba-d10019767ebf" (UID: "b0d71200-f415-498e-89ba-d10019767ebf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:48:03.403609 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.403466 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b0d71200-f415-498e-89ba-d10019767ebf" (UID: "b0d71200-f415-498e-89ba-d10019767ebf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:48:03.403968 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.403940 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-data" (OuterVolumeSpecName: "istio-data") pod "b0d71200-f415-498e-89ba-d10019767ebf" (UID: "b0d71200-f415-498e-89ba-d10019767ebf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:48:03.405029 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.405007 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0d71200-f415-498e-89ba-d10019767ebf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b0d71200-f415-498e-89ba-d10019767ebf" (UID: "b0d71200-f415-498e-89ba-d10019767ebf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 04:48:03.405201 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.405184 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-istio-token" (OuterVolumeSpecName: "istio-token") pod "b0d71200-f415-498e-89ba-d10019767ebf" (UID: "b0d71200-f415-498e-89ba-d10019767ebf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:48:03.405426 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.405398 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b0d71200-f415-498e-89ba-d10019767ebf" (UID: "b0d71200-f415-498e-89ba-d10019767ebf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:48:03.405487 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.405470 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-kube-api-access-5hnvz" (OuterVolumeSpecName: "kube-api-access-5hnvz") pod "b0d71200-f415-498e-89ba-d10019767ebf" (UID: "b0d71200-f415-498e-89ba-d10019767ebf"). InnerVolumeSpecName "kube-api-access-5hnvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:48:03.504610 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.504575 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:48:03.504610 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.504603 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-data\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:48:03.504610 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.504612 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-istio-envoy\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:48:03.504610 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.504620 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0d71200-f415-498e-89ba-d10019767ebf-istiod-ca-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:48:03.504960 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.504629 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hnvz\" (UniqueName: \"kubernetes.io/projected/b0d71200-f415-498e-89ba-d10019767ebf-kube-api-access-5hnvz\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:48:03.504960 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.504647 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0d71200-f415-498e-89ba-d10019767ebf-credential-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:48:03.504960 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:03.504655 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0d71200-f415-498e-89ba-d10019767ebf-istio-podinfo\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:48:04.221432 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:04.221394 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" event={"ID":"b0d71200-f415-498e-89ba-d10019767ebf","Type":"ContainerDied","Data":"09bc2503c7187dab6aa11c7beadc15eddffa6cfb20311aab702dc5e02e929f76"} Jul 21 04:48:04.221432 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:04.221425 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln" Jul 21 04:48:04.222045 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:04.221456 2567 scope.go:117] "RemoveContainer" containerID="811782f65acc22be105b113d369ec07403d8a07905bbe43f890b3dbbb9a489aa" Jul 21 04:48:04.246039 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:04.246017 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln"] Jul 21 04:48:04.251112 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:04.251090 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f63636-openshift-default-cd4c64655-tcmln"] Jul 21 04:48:05.267828 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:05.267795 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d71200-f415-498e-89ba-d10019767ebf" path="/var/lib/kubelet/pods/b0d71200-f415-498e-89ba-d10019767ebf/volumes" Jul 21 04:48:18.672025 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:18.671989 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:48:25.869535 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.869496 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47"] Jul 21 04:48:25.869994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.869934 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" Jul 21 04:48:25.869994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.869952 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" Jul 21 04:48:25.869994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.869964 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" Jul 21 04:48:25.869994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.869969 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" Jul 21 04:48:25.870134 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.870047 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" Jul 21 04:48:25.870134 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.870059 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d71200-f415-498e-89ba-d10019767ebf" containerName="istio-proxy" Jul 21 04:48:25.873214 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.873195 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:25.875557 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.875532 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-9fdb31-openshift-default-dockercfg-4ncsm\"" Jul 21 04:48:25.885052 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.885031 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47"] Jul 21 04:48:25.905473 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.905437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-credential-socket\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:25.905756 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.905731 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafc8dfb-7244-4c01-93de-ec36a77d4918-istiod-ca-cert\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:25.905910 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.905893 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-envoy\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:25.906062 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.906044 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hspk8\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-kube-api-access-hspk8\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:25.906196 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.906180 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-token\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:25.906320 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.906304 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-socket\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:25.906470 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.906451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-certs\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:25.906606 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.906588 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-data\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:25.906788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:25.906771 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-podinfo\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-podinfo\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008292 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-credential-socket\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008326 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafc8dfb-7244-4c01-93de-ec36a77d4918-istiod-ca-cert\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-envoy\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008412 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hspk8\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-kube-api-access-hspk8\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008446 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-token\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008471 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-socket\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008496 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-certs\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008587 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008522 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-data\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008983 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008795 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-credential-socket\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008983 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008868 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-socket\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.008983 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.008951 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-data\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.009161 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.009139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-certs\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.009215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.009163 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafc8dfb-7244-4c01-93de-ec36a77d4918-istiod-ca-cert\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.011326 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.011305 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-envoy\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.011502 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.011485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-podinfo\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.017575 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.017552 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-token\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.017575 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.017563 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hspk8\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-kube-api-access-hspk8\") pod \"e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.184483 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.184400 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:26.328003 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.327978 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47"] Jul 21 04:48:26.334506 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:48:26.334397 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafc8dfb_7244_4c01_93de_ec36a77d4918.slice/crio-fb54e200e23045a9a77aafef59574dc26bd25b00ebe4c1baa0935daafaac411f WatchSource:0}: Error finding container fb54e200e23045a9a77aafef59574dc26bd25b00ebe4c1baa0935daafaac411f: Status 404 returned error can't find the container with id fb54e200e23045a9a77aafef59574dc26bd25b00ebe4c1baa0935daafaac411f Jul 21 04:48:26.336554 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.336520 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:26.336673 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.336593 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:26.336673 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.336624 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:26.341551 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:26.341525 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" event={"ID":"dafc8dfb-7244-4c01-93de-ec36a77d4918","Type":"ContainerStarted","Data":"fb54e200e23045a9a77aafef59574dc26bd25b00ebe4c1baa0935daafaac411f"} Jul 21 04:48:27.348965 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:27.348927 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" event={"ID":"dafc8dfb-7244-4c01-93de-ec36a77d4918","Type":"ContainerStarted","Data":"0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57"} Jul 21 04:48:27.371185 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:27.371133 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" podStartSLOduration=2.371117171 podStartE2EDuration="2.371117171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:48:27.368053806 +0000 UTC m=+2352.767386556" watchObservedRunningTime="2026-07-21 04:48:27.371117171 +0000 UTC m=+2352.770449928" Jul 21 04:48:28.185010 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:28.184971 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:28.190381 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:28.190329 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:28.354327 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:28.354299 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:28.355399 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:28.355378 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:34.450091 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.450054 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6"] Jul 21 04:48:34.454136 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.454113 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.456368 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.456323 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-4de716-openshift-default-dockercfg-hjv26\"" Jul 21 04:48:34.466572 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.466542 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6"] Jul 21 04:48:34.490871 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.490843 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-credential-socket\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.491054 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.490875 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l4sh\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-kube-api-access-2l4sh\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.491054 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.490906 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-envoy\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.491054 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.491000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-data\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.491054 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.491038 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-podinfo\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.491248 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.491107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istiod-ca-cert\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.491248 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.491155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-certs\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.491248 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.491175 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-token\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.491248 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.491197 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-socket\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.592541 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.592506 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-credential-socket\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.592541 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.592542 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l4sh\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-kube-api-access-2l4sh\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.592829 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.592573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-envoy\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.592829 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.592591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-data\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.592829 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.592611 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-podinfo\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.592829 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.592641 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istiod-ca-cert\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.592829 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.592663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-certs\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.592829 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.592681 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-token\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.592829 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.592707 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-socket\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.593176 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.593038 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-data\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.593319 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.593292 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-certs\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.593489 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.593401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-socket\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.593489 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.593412 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-credential-socket\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.593614 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.593595 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istiod-ca-cert\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.595021 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.594996 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-envoy\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.595287 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.595271 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-podinfo\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.600931 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.600908 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-token\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.601028 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.600966 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l4sh\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-kube-api-access-2l4sh\") pod \"e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.768210 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.768178 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:34.918047 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.918019 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6"] Jul 21 04:48:34.920440 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:48:34.920406 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb2707a_dfcc_4712_8b10_9c395fef7b91.slice/crio-5b18a090e39ce3944c0d1633dab414b5bfac032551c00de780549dd576de65ff WatchSource:0}: Error finding container 5b18a090e39ce3944c0d1633dab414b5bfac032551c00de780549dd576de65ff: Status 404 returned error can't find the container with id 5b18a090e39ce3944c0d1633dab414b5bfac032551c00de780549dd576de65ff Jul 21 04:48:34.923125 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.923086 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:34.923228 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.923172 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:34.923228 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:34.923215 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:35.393186 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:35.393149 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" event={"ID":"bdb2707a-dfcc-4712-8b10-9c395fef7b91","Type":"ContainerStarted","Data":"4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e"} Jul 21 04:48:35.393478 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:35.393208 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" event={"ID":"bdb2707a-dfcc-4712-8b10-9c395fef7b91","Type":"ContainerStarted","Data":"5b18a090e39ce3944c0d1633dab414b5bfac032551c00de780549dd576de65ff"} Jul 21 04:48:35.416039 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:35.415986 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" podStartSLOduration=1.415970239 podStartE2EDuration="1.415970239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:48:35.414062983 +0000 UTC m=+2360.813395742" watchObservedRunningTime="2026-07-21 04:48:35.415970239 +0000 UTC m=+2360.815302997" Jul 21 04:48:35.768889 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:35.768855 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:35.773686 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:35.773664 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:36.398911 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:36.398875 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:36.399912 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:36.399891 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:36.877259 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:36.877228 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:48:44.396499 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:44.396458 2567 patch_prober.go:28] interesting pod/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 04:48:44.396902 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:44.396518 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 21 04:48:45.456149 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:45.456117 2567 generic.go:356] "Generic (PLEG): container finished" podID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerID="0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57" exitCode=0 Jul 21 04:48:45.456768 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:45.456173 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" event={"ID":"dafc8dfb-7244-4c01-93de-ec36a77d4918","Type":"ContainerDied","Data":"0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57"} Jul 21 04:48:45.456768 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:45.456724 2567 scope.go:117] "RemoveContainer" containerID="0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57" Jul 21 04:48:45.457774 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:45.457737 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:45.457877 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:45.457812 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:45.457877 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:45.457843 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:46.470099 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:46.470054 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" event={"ID":"dafc8dfb-7244-4c01-93de-ec36a77d4918","Type":"ContainerStarted","Data":"15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc"} Jul 21 04:48:47.185319 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:47.185290 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:47.185590 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:47.185458 2567 patch_prober.go:28] interesting pod/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 21 04:48:47.185709 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:47.185692 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 21 04:48:48.185738 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:48.185699 2567 patch_prober.go:28] interesting pod/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 21 04:48:48.186167 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:48.185762 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 21 04:48:49.185322 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:49.185279 2567 patch_prober.go:28] interesting pod/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 21 04:48:49.185535 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:49.185360 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 21 04:48:50.185324 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:50.185287 2567 patch_prober.go:28] interesting pod/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 21 04:48:50.185728 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:50.185373 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 21 04:48:51.176625 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:51.176582 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:48:51.190328 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:51.190288 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:51.190778 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:51.190721 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:51.191635 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:51.191610 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:48:52.441529 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:52.441485 2567 patch_prober.go:28] interesting pod/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 04:48:52.442000 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:52.441555 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 21 04:48:53.513702 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:53.513673 2567 generic.go:356] "Generic (PLEG): container finished" podID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerID="4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e" exitCode=0 Jul 21 04:48:53.514126 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:53.513750 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" event={"ID":"bdb2707a-dfcc-4712-8b10-9c395fef7b91","Type":"ContainerDied","Data":"4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e"} Jul 21 04:48:53.514169 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:53.514146 2567 scope.go:117] "RemoveContainer" containerID="4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e" Jul 21 04:48:53.515111 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:53.515085 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:53.515169 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:53.515149 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:53.515204 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:53.515182 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 21 04:48:54.524274 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:54.524239 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" event={"ID":"bdb2707a-dfcc-4712-8b10-9c395fef7b91","Type":"ContainerStarted","Data":"9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405"} Jul 21 04:48:54.769251 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:54.769211 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:54.769543 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:54.769520 2567 patch_prober.go:28] interesting pod/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 21 04:48:54.769595 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:54.769582 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 21 04:48:55.768839 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:55.768798 2567 patch_prober.go:28] interesting pod/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 21 04:48:55.769241 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:55.768864 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 21 04:48:56.768585 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:56.768551 2567 patch_prober.go:28] interesting pod/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 21 04:48:56.768845 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:56.768613 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 21 04:48:57.768508 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:57.768473 2567 patch_prober.go:28] interesting pod/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 21 04:48:57.768929 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:57.768529 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 21 04:48:58.772845 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:58.772815 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:58.773260 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:58.773239 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:48:58.773877 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:48:58.773848 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:49:15.265655 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:15.265628 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:49:15.272780 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:15.272756 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:49:15.987187 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:15.987145 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm"] Jul 21 04:49:15.991114 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:15.991087 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:15.993720 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:15.993672 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9fdb31\"/\"openshift-service-ca.crt\"" Jul 21 04:49:15.994764 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:15.994744 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9fdb31\"/\"default-dockercfg-cn5fg\"" Jul 21 04:49:15.994927 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:15.994906 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9fdb31\"/\"auth-test-model-9fdb31-kserve-self-signed-certs\"" Jul 21 04:49:15.995090 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:15.994749 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9fdb31\"/\"kube-root-ca.crt\"" Jul 21 04:49:16.000404 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.000379 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm"] Jul 21 04:49:16.097649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.097612 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-tmp-dir\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.097649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.097656 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-dshm\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.097914 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.097678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzghk\" (UniqueName: \"kubernetes.io/projected/56db9e04-d117-4882-bcf0-9b3eea843ebd-kube-api-access-vzghk\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.097914 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.097799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-model-cache\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.097914 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.097852 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-home\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.098048 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.097931 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56db9e04-d117-4882-bcf0-9b3eea843ebd-tls-certs\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.198414 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.198371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-home\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.198616 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.198422 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56db9e04-d117-4882-bcf0-9b3eea843ebd-tls-certs\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.198616 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.198478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-tmp-dir\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.198616 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.198505 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-dshm\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.198616 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.198531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzghk\" (UniqueName: \"kubernetes.io/projected/56db9e04-d117-4882-bcf0-9b3eea843ebd-kube-api-access-vzghk\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.198850 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.198627 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-model-cache\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.198911 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.198863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-home\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.198972 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.198896 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-tmp-dir\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.199030 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.198979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-model-cache\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.200840 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.200809 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-dshm\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.201027 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.201011 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56db9e04-d117-4882-bcf0-9b3eea843ebd-tls-certs\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.207379 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.207357 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzghk\" (UniqueName: \"kubernetes.io/projected/56db9e04-d117-4882-bcf0-9b3eea843ebd-kube-api-access-vzghk\") pod \"auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.304268 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.304236 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:16.444718 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.444691 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm"] Jul 21 04:49:16.447286 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:49:16.447258 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56db9e04_d117_4882_bcf0_9b3eea843ebd.slice/crio-b82cf7d71e9b45e06684ce9ed85b722135929ea190e400cc2ddd633880fb026a WatchSource:0}: Error finding container b82cf7d71e9b45e06684ce9ed85b722135929ea190e400cc2ddd633880fb026a: Status 404 returned error can't find the container with id b82cf7d71e9b45e06684ce9ed85b722135929ea190e400cc2ddd633880fb026a Jul 21 04:49:16.656970 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.656864 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" event={"ID":"56db9e04-d117-4882-bcf0-9b3eea843ebd","Type":"ContainerStarted","Data":"457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a"} Jul 21 04:49:16.656970 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.656911 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" event={"ID":"56db9e04-d117-4882-bcf0-9b3eea843ebd","Type":"ContainerStarted","Data":"b82cf7d71e9b45e06684ce9ed85b722135929ea190e400cc2ddd633880fb026a"} Jul 21 04:49:16.674093 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:16.674042 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" podStartSLOduration=1.674024289 podStartE2EDuration="1.674024289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:49:16.672028658 +0000 UTC m=+2402.071361416" watchObservedRunningTime="2026-07-21 04:49:16.674024289 +0000 UTC m=+2402.073357068" Jul 21 04:49:19.579478 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:19.579437 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:49:26.305008 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:26.304965 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:26.305472 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:26.305050 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:26.317659 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:26.317637 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:26.729949 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:26.729862 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:49:34.769034 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:34.769000 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:49:59.183738 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:49:59.183656 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:50:17.384712 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:50:17.384679 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:50:32.788569 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:50:32.788535 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:50:52.785584 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:50:52.785546 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:51:09.979239 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:09.979206 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:51:23.776280 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:23.776240 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:51:39.084032 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:39.083998 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm"] Jul 21 04:51:41.484547 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:41.484517 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:51:54.104234 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.104171 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" podUID="56db9e04-d117-4882-bcf0-9b3eea843ebd" containerName="main" containerID="cri-o://457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a" gracePeriod=45 Jul 21 04:51:54.373851 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.373825 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:51:54.387787 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.387764 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzghk\" (UniqueName: \"kubernetes.io/projected/56db9e04-d117-4882-bcf0-9b3eea843ebd-kube-api-access-vzghk\") pod \"56db9e04-d117-4882-bcf0-9b3eea843ebd\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " Jul 21 04:51:54.387881 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.387800 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-model-cache\") pod \"56db9e04-d117-4882-bcf0-9b3eea843ebd\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " Jul 21 04:51:54.387881 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.387824 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-tmp-dir\") pod \"56db9e04-d117-4882-bcf0-9b3eea843ebd\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " Jul 21 04:51:54.387952 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.387898 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56db9e04-d117-4882-bcf0-9b3eea843ebd-tls-certs\") pod \"56db9e04-d117-4882-bcf0-9b3eea843ebd\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " Jul 21 04:51:54.387994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.387966 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-dshm\") pod \"56db9e04-d117-4882-bcf0-9b3eea843ebd\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " Jul 21 04:51:54.388045 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.388013 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-home\") pod \"56db9e04-d117-4882-bcf0-9b3eea843ebd\" (UID: \"56db9e04-d117-4882-bcf0-9b3eea843ebd\") " Jul 21 04:51:54.388101 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.388052 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-model-cache" (OuterVolumeSpecName: "model-cache") pod "56db9e04-d117-4882-bcf0-9b3eea843ebd" (UID: "56db9e04-d117-4882-bcf0-9b3eea843ebd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:51:54.388160 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.388094 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56db9e04-d117-4882-bcf0-9b3eea843ebd" (UID: "56db9e04-d117-4882-bcf0-9b3eea843ebd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:51:54.388352 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.388302 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-home" (OuterVolumeSpecName: "home") pod "56db9e04-d117-4882-bcf0-9b3eea843ebd" (UID: "56db9e04-d117-4882-bcf0-9b3eea843ebd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:51:54.388352 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.388320 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-tmp-dir\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:51:54.388352 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.388357 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-model-cache\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:51:54.390066 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.390040 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-dshm" (OuterVolumeSpecName: "dshm") pod "56db9e04-d117-4882-bcf0-9b3eea843ebd" (UID: "56db9e04-d117-4882-bcf0-9b3eea843ebd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:51:54.390155 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.390032 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56db9e04-d117-4882-bcf0-9b3eea843ebd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56db9e04-d117-4882-bcf0-9b3eea843ebd" (UID: "56db9e04-d117-4882-bcf0-9b3eea843ebd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:51:54.390155 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.390126 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56db9e04-d117-4882-bcf0-9b3eea843ebd-kube-api-access-vzghk" (OuterVolumeSpecName: "kube-api-access-vzghk") pod "56db9e04-d117-4882-bcf0-9b3eea843ebd" (UID: "56db9e04-d117-4882-bcf0-9b3eea843ebd"). InnerVolumeSpecName "kube-api-access-vzghk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:51:54.489422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.489329 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-dshm\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:51:54.489422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.489368 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56db9e04-d117-4882-bcf0-9b3eea843ebd-home\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:51:54.489422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.489381 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzghk\" (UniqueName: \"kubernetes.io/projected/56db9e04-d117-4882-bcf0-9b3eea843ebd-kube-api-access-vzghk\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:51:54.489422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.489395 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56db9e04-d117-4882-bcf0-9b3eea843ebd-tls-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:51:54.555684 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.555653 2567 generic.go:356] "Generic (PLEG): container finished" podID="56db9e04-d117-4882-bcf0-9b3eea843ebd" containerID="457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a" exitCode=0 Jul 21 04:51:54.555849 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.555698 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" Jul 21 04:51:54.555849 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.555744 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" event={"ID":"56db9e04-d117-4882-bcf0-9b3eea843ebd","Type":"ContainerDied","Data":"457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a"} Jul 21 04:51:54.555849 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.555787 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm" event={"ID":"56db9e04-d117-4882-bcf0-9b3eea843ebd","Type":"ContainerDied","Data":"b82cf7d71e9b45e06684ce9ed85b722135929ea190e400cc2ddd633880fb026a"} Jul 21 04:51:54.555849 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.555809 2567 scope.go:117] "RemoveContainer" containerID="457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a" Jul 21 04:51:54.565755 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.565735 2567 scope.go:117] "RemoveContainer" containerID="457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a" Jul 21 04:51:54.566011 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:51:54.565990 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a\": container with ID starting with 457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a not found: ID does not exist" containerID="457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a" Jul 21 04:51:54.566056 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.566019 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a"} err="failed to get container status \"457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a\": rpc error: code = NotFound desc = could not find container \"457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a\": container with ID starting with 457747d1e6512bd38dd5554c71e396f064676a980f6a2b8920d97a2e2c91984a not found: ID does not exist" Jul 21 04:51:54.579185 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.579162 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm"] Jul 21 04:51:54.583152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:54.583134 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/auth-test-model-9fdb31-kserve-66f8cf7c5-wn9rm"] Jul 21 04:51:55.267765 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:55.267726 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56db9e04-d117-4882-bcf0-9b3eea843ebd" path="/var/lib/kubelet/pods/56db9e04-d117-4882-bcf0-9b3eea843ebd/volumes" Jul 21 04:51:56.284547 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:51:56.284513 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:52:18.388309 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.388272 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g"] Jul 21 04:52:18.388741 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.388702 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56db9e04-d117-4882-bcf0-9b3eea843ebd" containerName="main" Jul 21 04:52:18.388741 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.388719 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="56db9e04-d117-4882-bcf0-9b3eea843ebd" containerName="main" Jul 21 04:52:18.388831 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.388789 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="56db9e04-d117-4882-bcf0-9b3eea843ebd" containerName="main" Jul 21 04:52:18.393579 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.393559 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.396854 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.396835 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4de716\"/\"kube-root-ca.crt\"" Jul 21 04:52:18.397783 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.397741 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4de716\"/\"openshift-service-ca.crt\"" Jul 21 04:52:18.397910 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.397748 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4de716\"/\"default-dockercfg-c7hw8\"" Jul 21 04:52:18.398067 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.398049 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4de716\"/\"sub-test-model-4de716-kserve-self-signed-certs\"" Jul 21 04:52:18.402507 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.402488 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g"] Jul 21 04:52:18.505256 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.505227 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-home\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.505448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.505264 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tls-certs\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.505448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.505364 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-model-cache\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.505448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.505395 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tmp-dir\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.505448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.505418 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjl9\" (UniqueName: \"kubernetes.io/projected/934fbf76-9869-4dec-80ce-e6d913fc5ac1-kube-api-access-kcjl9\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.505448 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.505437 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-dshm\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.606832 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.606796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-model-cache\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.606832 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.606835 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tmp-dir\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.607095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.606859 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjl9\" (UniqueName: \"kubernetes.io/projected/934fbf76-9869-4dec-80ce-e6d913fc5ac1-kube-api-access-kcjl9\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.607095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.606880 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-dshm\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.607095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.606915 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-home\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.607095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.606942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tls-certs\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.607314 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.607281 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-model-cache\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.607314 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.607288 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tmp-dir\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.607463 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.607406 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-home\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.609328 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.609305 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-dshm\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.609560 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.609544 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tls-certs\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.616785 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.616761 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjl9\" (UniqueName: \"kubernetes.io/projected/934fbf76-9869-4dec-80ce-e6d913fc5ac1-kube-api-access-kcjl9\") pod \"sub-test-model-4de716-kserve-57bbf76bfc-x2g6g\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.704025 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.703955 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:18.842825 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:18.842797 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g"] Jul 21 04:52:18.844136 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:52:18.844109 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod934fbf76_9869_4dec_80ce_e6d913fc5ac1.slice/crio-619d062ff54f156de8b80ca400f0d22e0896420ca195525a67299551bdc7f98e WatchSource:0}: Error finding container 619d062ff54f156de8b80ca400f0d22e0896420ca195525a67299551bdc7f98e: Status 404 returned error can't find the container with id 619d062ff54f156de8b80ca400f0d22e0896420ca195525a67299551bdc7f98e Jul 21 04:52:19.698314 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:19.698277 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" event={"ID":"934fbf76-9869-4dec-80ce-e6d913fc5ac1","Type":"ContainerStarted","Data":"26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e"} Jul 21 04:52:19.698314 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:19.698321 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" event={"ID":"934fbf76-9869-4dec-80ce-e6d913fc5ac1","Type":"ContainerStarted","Data":"619d062ff54f156de8b80ca400f0d22e0896420ca195525a67299551bdc7f98e"} Jul 21 04:52:19.715073 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:19.715027 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" podStartSLOduration=1.715011546 podStartE2EDuration="1.715011546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:52:19.714592953 +0000 UTC m=+2585.113925711" watchObservedRunningTime="2026-07-21 04:52:19.715011546 +0000 UTC m=+2585.114344304" Jul 21 04:52:22.497367 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:22.495209 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:52:28.704633 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:28.704597 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:28.705049 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:28.704654 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:28.717431 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:28.717405 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:52:28.761502 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:52:28.761473 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:53:12.983987 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:12.983957 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:53:33.373999 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:33.373960 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:53:39.369664 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:39.369627 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g"] Jul 21 04:53:51.778921 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:51.778885 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:53:54.393528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.393487 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" podUID="934fbf76-9869-4dec-80ce-e6d913fc5ac1" containerName="main" containerID="cri-o://26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e" gracePeriod=45 Jul 21 04:53:54.649177 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.649108 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:53:54.750636 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.750604 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tmp-dir\") pod \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " Jul 21 04:53:54.750813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.750661 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tls-certs\") pod \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " Jul 21 04:53:54.750813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.750698 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcjl9\" (UniqueName: \"kubernetes.io/projected/934fbf76-9869-4dec-80ce-e6d913fc5ac1-kube-api-access-kcjl9\") pod \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " Jul 21 04:53:54.750813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.750720 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-dshm\") pod \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " Jul 21 04:53:54.750813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.750792 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-home\") pod \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " Jul 21 04:53:54.751017 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.750847 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-model-cache\") pod \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\" (UID: \"934fbf76-9869-4dec-80ce-e6d913fc5ac1\") " Jul 21 04:53:54.751017 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.750903 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "934fbf76-9869-4dec-80ce-e6d913fc5ac1" (UID: "934fbf76-9869-4dec-80ce-e6d913fc5ac1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:53:54.751133 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.751087 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-home" (OuterVolumeSpecName: "home") pod "934fbf76-9869-4dec-80ce-e6d913fc5ac1" (UID: "934fbf76-9869-4dec-80ce-e6d913fc5ac1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:53:54.751191 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.751141 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-model-cache" (OuterVolumeSpecName: "model-cache") pod "934fbf76-9869-4dec-80ce-e6d913fc5ac1" (UID: "934fbf76-9869-4dec-80ce-e6d913fc5ac1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:53:54.751191 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.751163 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tmp-dir\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:53:54.751191 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.751183 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-home\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:53:54.752954 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.752933 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-dshm" (OuterVolumeSpecName: "dshm") pod "934fbf76-9869-4dec-80ce-e6d913fc5ac1" (UID: "934fbf76-9869-4dec-80ce-e6d913fc5ac1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:53:54.753060 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.753041 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934fbf76-9869-4dec-80ce-e6d913fc5ac1-kube-api-access-kcjl9" (OuterVolumeSpecName: "kube-api-access-kcjl9") pod "934fbf76-9869-4dec-80ce-e6d913fc5ac1" (UID: "934fbf76-9869-4dec-80ce-e6d913fc5ac1"). InnerVolumeSpecName "kube-api-access-kcjl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:53:54.753100 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.753060 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "934fbf76-9869-4dec-80ce-e6d913fc5ac1" (UID: "934fbf76-9869-4dec-80ce-e6d913fc5ac1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:53:54.851980 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.851950 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcjl9\" (UniqueName: \"kubernetes.io/projected/934fbf76-9869-4dec-80ce-e6d913fc5ac1-kube-api-access-kcjl9\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:53:54.851980 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.851975 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-dshm\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:53:54.852146 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.851989 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/934fbf76-9869-4dec-80ce-e6d913fc5ac1-model-cache\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:53:54.852146 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:54.852003 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934fbf76-9869-4dec-80ce-e6d913fc5ac1-tls-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:53:55.256222 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:55.256189 2567 generic.go:356] "Generic (PLEG): container finished" podID="934fbf76-9869-4dec-80ce-e6d913fc5ac1" containerID="26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e" exitCode=0 Jul 21 04:53:55.256222 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:55.256227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" event={"ID":"934fbf76-9869-4dec-80ce-e6d913fc5ac1","Type":"ContainerDied","Data":"26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e"} Jul 21 04:53:55.256483 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:55.256234 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" Jul 21 04:53:55.256483 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:55.256258 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g" event={"ID":"934fbf76-9869-4dec-80ce-e6d913fc5ac1","Type":"ContainerDied","Data":"619d062ff54f156de8b80ca400f0d22e0896420ca195525a67299551bdc7f98e"} Jul 21 04:53:55.256483 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:55.256275 2567 scope.go:117] "RemoveContainer" containerID="26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e" Jul 21 04:53:55.266999 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:55.266979 2567 scope.go:117] "RemoveContainer" containerID="26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e" Jul 21 04:53:55.267240 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:53:55.267219 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e\": container with ID starting with 26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e not found: ID does not exist" containerID="26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e" Jul 21 04:53:55.267288 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:55.267248 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e"} err="failed to get container status \"26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e\": rpc error: code = NotFound desc = could not find container \"26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e\": container with ID starting with 26d4ea937b38c420823923f791c919312c8417a36f9ab75cad31d400c24bb80e not found: ID does not exist" Jul 21 04:53:55.279307 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:55.279284 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g"] Jul 21 04:53:55.283673 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:55.283640 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4de716/sub-test-model-4de716-kserve-57bbf76bfc-x2g6g"] Jul 21 04:53:57.267583 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:53:57.267549 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934fbf76-9869-4dec-80ce-e6d913fc5ac1" path="/var/lib/kubelet/pods/934fbf76-9869-4dec-80ce-e6d913fc5ac1/volumes" Jul 21 04:54:02.490502 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.490469 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7"] Jul 21 04:54:02.491085 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.491064 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934fbf76-9869-4dec-80ce-e6d913fc5ac1" containerName="main" Jul 21 04:54:02.491179 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.491088 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="934fbf76-9869-4dec-80ce-e6d913fc5ac1" containerName="main" Jul 21 04:54:02.491236 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.491220 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="934fbf76-9869-4dec-80ce-e6d913fc5ac1" containerName="main" Jul 21 04:54:02.495393 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.495372 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.499143 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.499115 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9fdb31\"/\"openshift-service-ca.crt\"" Jul 21 04:54:02.499361 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.499122 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9fdb31\"/\"kube-root-ca.crt\"" Jul 21 04:54:02.499462 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.499415 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9fdb31\"/\"rate-test-model-9fdb31-kserve-self-signed-certs\"" Jul 21 04:54:02.499616 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.499599 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9fdb31\"/\"default-dockercfg-cn5fg\"" Jul 21 04:54:02.503529 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.503188 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7"] Jul 21 04:54:02.624960 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.624914 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btvc7\" (UniqueName: \"kubernetes.io/projected/c64cb7e9-eece-4205-ad8b-89a1dd121eae-kube-api-access-btvc7\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.625167 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.625031 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-home\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.625167 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.625084 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tmp-dir\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.625167 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.625142 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-dshm\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.625353 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.625168 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tls-certs\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.625353 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.625279 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-model-cache\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.726322 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.726282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-model-cache\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.726513 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.726357 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btvc7\" (UniqueName: \"kubernetes.io/projected/c64cb7e9-eece-4205-ad8b-89a1dd121eae-kube-api-access-btvc7\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.726513 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.726417 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-home\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.726513 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.726451 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tmp-dir\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.726627 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.726590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-dshm\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.726683 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.726636 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tls-certs\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.726808 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.726784 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-model-cache\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.726858 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.726808 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tmp-dir\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.726934 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.726914 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-home\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.728965 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.728943 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-dshm\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.729238 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.729220 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tls-certs\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.735733 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.735712 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btvc7\" (UniqueName: \"kubernetes.io/projected/c64cb7e9-eece-4205-ad8b-89a1dd121eae-kube-api-access-btvc7\") pod \"rate-test-model-9fdb31-kserve-68fd895495-27pj7\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.806604 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.806566 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:02.949254 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:02.949223 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7"] Jul 21 04:54:02.950838 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:54:02.950812 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc64cb7e9_eece_4205_ad8b_89a1dd121eae.slice/crio-e04341a7845ca84d5d02f7980007f0ad515db54bb232cfc69ec8b05a988b09b5 WatchSource:0}: Error finding container e04341a7845ca84d5d02f7980007f0ad515db54bb232cfc69ec8b05a988b09b5: Status 404 returned error can't find the container with id e04341a7845ca84d5d02f7980007f0ad515db54bb232cfc69ec8b05a988b09b5 Jul 21 04:54:03.303012 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:03.302972 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" event={"ID":"c64cb7e9-eece-4205-ad8b-89a1dd121eae","Type":"ContainerStarted","Data":"e7f43b0b3ef458f1ab8d0d4cb90f67e78474f6c536b4c7d2b68dd7a90377c2e4"} Jul 21 04:54:03.303160 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:03.303027 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" event={"ID":"c64cb7e9-eece-4205-ad8b-89a1dd121eae","Type":"ContainerStarted","Data":"e04341a7845ca84d5d02f7980007f0ad515db54bb232cfc69ec8b05a988b09b5"} Jul 21 04:54:03.321404 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:03.321352 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" podStartSLOduration=1.321323711 podStartE2EDuration="1.321323711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:54:03.318963216 +0000 UTC m=+2688.718295975" watchObservedRunningTime="2026-07-21 04:54:03.321323711 +0000 UTC m=+2688.720656459" Jul 21 04:54:06.079196 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:06.079155 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:54:12.807548 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:12.807504 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:12.807939 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:12.807572 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:12.820254 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:12.820222 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:13.371084 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:13.371052 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:54:15.301716 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:15.301688 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:54:15.309291 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:15.309260 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:54:34.682886 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:34.682842 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:54:49.181474 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:54:49.181443 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:55:04.776947 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:04.776913 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:55:11.565010 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:11.564974 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7"] Jul 21 04:55:19.075322 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:19.075284 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:55:26.585652 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.585610 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" podUID="c64cb7e9-eece-4205-ad8b-89a1dd121eae" containerName="main" containerID="cri-o://e7f43b0b3ef458f1ab8d0d4cb90f67e78474f6c536b4c7d2b68dd7a90377c2e4" gracePeriod=45 Jul 21 04:55:26.787150 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.787120 2567 generic.go:356] "Generic (PLEG): container finished" podID="c64cb7e9-eece-4205-ad8b-89a1dd121eae" containerID="e7f43b0b3ef458f1ab8d0d4cb90f67e78474f6c536b4c7d2b68dd7a90377c2e4" exitCode=0 Jul 21 04:55:26.787303 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.787191 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" event={"ID":"c64cb7e9-eece-4205-ad8b-89a1dd121eae","Type":"ContainerDied","Data":"e7f43b0b3ef458f1ab8d0d4cb90f67e78474f6c536b4c7d2b68dd7a90377c2e4"} Jul 21 04:55:26.845607 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.845549 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:55:26.915533 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.915504 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tmp-dir\") pod \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " Jul 21 04:55:26.915703 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.915540 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btvc7\" (UniqueName: \"kubernetes.io/projected/c64cb7e9-eece-4205-ad8b-89a1dd121eae-kube-api-access-btvc7\") pod \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " Jul 21 04:55:26.915703 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.915564 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-model-cache\") pod \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " Jul 21 04:55:26.915703 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.915601 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-home\") pod \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " Jul 21 04:55:26.915703 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.915638 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tls-certs\") pod \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " Jul 21 04:55:26.915912 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.915704 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-dshm\") pod \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\" (UID: \"c64cb7e9-eece-4205-ad8b-89a1dd121eae\") " Jul 21 04:55:26.915912 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.915828 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c64cb7e9-eece-4205-ad8b-89a1dd121eae" (UID: "c64cb7e9-eece-4205-ad8b-89a1dd121eae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:55:26.915912 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.915812 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-model-cache" (OuterVolumeSpecName: "model-cache") pod "c64cb7e9-eece-4205-ad8b-89a1dd121eae" (UID: "c64cb7e9-eece-4205-ad8b-89a1dd121eae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:55:26.915912 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.915877 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-home" (OuterVolumeSpecName: "home") pod "c64cb7e9-eece-4205-ad8b-89a1dd121eae" (UID: "c64cb7e9-eece-4205-ad8b-89a1dd121eae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:55:26.916110 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.916075 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tmp-dir\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:26.916110 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.916091 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-model-cache\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:26.916110 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.916098 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-home\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:26.917676 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.917650 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64cb7e9-eece-4205-ad8b-89a1dd121eae-kube-api-access-btvc7" (OuterVolumeSpecName: "kube-api-access-btvc7") pod "c64cb7e9-eece-4205-ad8b-89a1dd121eae" (UID: "c64cb7e9-eece-4205-ad8b-89a1dd121eae"). InnerVolumeSpecName "kube-api-access-btvc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:55:26.917852 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.917835 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c64cb7e9-eece-4205-ad8b-89a1dd121eae" (UID: "c64cb7e9-eece-4205-ad8b-89a1dd121eae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 04:55:26.917852 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:26.917842 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-dshm" (OuterVolumeSpecName: "dshm") pod "c64cb7e9-eece-4205-ad8b-89a1dd121eae" (UID: "c64cb7e9-eece-4205-ad8b-89a1dd121eae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:55:27.016788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:27.016738 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c64cb7e9-eece-4205-ad8b-89a1dd121eae-dshm\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:27.016788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:27.016784 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btvc7\" (UniqueName: \"kubernetes.io/projected/c64cb7e9-eece-4205-ad8b-89a1dd121eae-kube-api-access-btvc7\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:27.016788 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:27.016794 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c64cb7e9-eece-4205-ad8b-89a1dd121eae-tls-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:27.794043 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:27.794018 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" Jul 21 04:55:27.794525 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:27.794056 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7" event={"ID":"c64cb7e9-eece-4205-ad8b-89a1dd121eae","Type":"ContainerDied","Data":"e04341a7845ca84d5d02f7980007f0ad515db54bb232cfc69ec8b05a988b09b5"} Jul 21 04:55:27.794525 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:27.794101 2567 scope.go:117] "RemoveContainer" containerID="e7f43b0b3ef458f1ab8d0d4cb90f67e78474f6c536b4c7d2b68dd7a90377c2e4" Jul 21 04:55:27.813269 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:27.813243 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7"] Jul 21 04:55:27.817508 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:27.817484 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9fdb31/rate-test-model-9fdb31-kserve-68fd895495-27pj7"] Jul 21 04:55:29.048185 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.048152 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7799-140183462626768"] Jul 21 04:55:29.048594 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.048583 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64cb7e9-eece-4205-ad8b-89a1dd121eae" containerName="main" Jul 21 04:55:29.048647 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.048596 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64cb7e9-eece-4205-ad8b-89a1dd121eae" containerName="main" Jul 21 04:55:29.048685 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.048672 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c64cb7e9-eece-4205-ad8b-89a1dd121eae" containerName="main" Jul 21 04:55:29.053015 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.052997 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7799-140183462626768" Jul 21 04:55:29.056197 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.056174 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-t7rfl\"" Jul 21 04:55:29.058994 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.058970 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7799-140183462626768"] Jul 21 04:55:29.135386 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.135360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbvs\" (UniqueName: \"kubernetes.io/projected/cd32518d-8a29-4151-9229-a159ab19fad0-kube-api-access-msbvs\") pod \"test-curl-7799-140183462626768\" (UID: \"cd32518d-8a29-4151-9229-a159ab19fad0\") " pod="opendatahub/test-curl-7799-140183462626768" Jul 21 04:55:29.236841 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.236804 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbvs\" (UniqueName: \"kubernetes.io/projected/cd32518d-8a29-4151-9229-a159ab19fad0-kube-api-access-msbvs\") pod \"test-curl-7799-140183462626768\" (UID: \"cd32518d-8a29-4151-9229-a159ab19fad0\") " pod="opendatahub/test-curl-7799-140183462626768" Jul 21 04:55:29.245800 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.245770 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbvs\" (UniqueName: \"kubernetes.io/projected/cd32518d-8a29-4151-9229-a159ab19fad0-kube-api-access-msbvs\") pod \"test-curl-7799-140183462626768\" (UID: \"cd32518d-8a29-4151-9229-a159ab19fad0\") " pod="opendatahub/test-curl-7799-140183462626768" Jul 21 04:55:29.267105 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.267080 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c64cb7e9-eece-4205-ad8b-89a1dd121eae" path="/var/lib/kubelet/pods/c64cb7e9-eece-4205-ad8b-89a1dd121eae/volumes" Jul 21 04:55:29.364422 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.364358 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7799-140183462626768" Jul 21 04:55:29.504380 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.502040 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7799-140183462626768"] Jul 21 04:55:29.504380 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.502501 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 04:55:29.808377 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:29.808323 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7799-140183462626768" event={"ID":"cd32518d-8a29-4151-9229-a159ab19fad0","Type":"ContainerStarted","Data":"5a7fca74baaaa9088cbce9a5e6e57c3a78bfa240bc081cc5d8dcd559b6ee0152"} Jul 21 04:55:30.815114 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:30.815077 2567 generic.go:356] "Generic (PLEG): container finished" podID="cd32518d-8a29-4151-9229-a159ab19fad0" containerID="edc2a702bc47af87185d2b27db92a0c6d7eab3a5baca398960bf82a0df1f7754" exitCode=0 Jul 21 04:55:30.815597 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:30.815161 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7799-140183462626768" event={"ID":"cd32518d-8a29-4151-9229-a159ab19fad0","Type":"ContainerDied","Data":"edc2a702bc47af87185d2b27db92a0c6d7eab3a5baca398960bf82a0df1f7754"} Jul 21 04:55:31.955936 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:31.955912 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7799-140183462626768" Jul 21 04:55:31.993330 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:31.993305 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7799-140183462626768_cd32518d-8a29-4151-9229-a159ab19fad0/test-curl-7799-140183462626768/0.log" Jul 21 04:55:32.015717 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:32.015692 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7799-140183462626768"] Jul 21 04:55:32.018027 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:32.018002 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7799-140183462626768"] Jul 21 04:55:32.063794 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:32.063771 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msbvs\" (UniqueName: \"kubernetes.io/projected/cd32518d-8a29-4151-9229-a159ab19fad0-kube-api-access-msbvs\") pod \"cd32518d-8a29-4151-9229-a159ab19fad0\" (UID: \"cd32518d-8a29-4151-9229-a159ab19fad0\") " Jul 21 04:55:32.066217 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:32.066195 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd32518d-8a29-4151-9229-a159ab19fad0-kube-api-access-msbvs" (OuterVolumeSpecName: "kube-api-access-msbvs") pod "cd32518d-8a29-4151-9229-a159ab19fad0" (UID: "cd32518d-8a29-4151-9229-a159ab19fad0"). InnerVolumeSpecName "kube-api-access-msbvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:55:32.164924 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:32.164901 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msbvs\" (UniqueName: \"kubernetes.io/projected/cd32518d-8a29-4151-9229-a159ab19fad0-kube-api-access-msbvs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:32.829460 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:32.829431 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a7fca74baaaa9088cbce9a5e6e57c3a78bfa240bc081cc5d8dcd559b6ee0152" Jul 21 04:55:32.829686 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:32.829665 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7799-140183462626768" Jul 21 04:55:32.840533 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:55:32.840498 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7799-140183462626768\" is forbidden: User \"system:node:ip-10-0-137-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-226.ec2.internal' and this object" podUID="cd32518d-8a29-4151-9229-a159ab19fad0" pod="opendatahub/test-curl-7799-140183462626768" Jul 21 04:55:33.266726 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:33.266696 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd32518d-8a29-4151-9229-a159ab19fad0" path="/var/lib/kubelet/pods/cd32518d-8a29-4151-9229-a159ab19fad0/volumes" Jul 21 04:55:45.574726 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:45.574647 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:55:54.025586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:54.025548 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47"] Jul 21 04:55:54.026058 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:54.025806 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" containerID="cri-o://15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc" gracePeriod=30 Jul 21 04:55:59.278596 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.278570 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:55:59.306033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.305967 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-envoy\") pod \"dafc8dfb-7244-4c01-93de-ec36a77d4918\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " Jul 21 04:55:59.306033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306024 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafc8dfb-7244-4c01-93de-ec36a77d4918-istiod-ca-cert\") pod \"dafc8dfb-7244-4c01-93de-ec36a77d4918\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " Jul 21 04:55:59.306207 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306057 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-certs\") pod \"dafc8dfb-7244-4c01-93de-ec36a77d4918\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " Jul 21 04:55:59.306207 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306113 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-podinfo\") pod \"dafc8dfb-7244-4c01-93de-ec36a77d4918\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " Jul 21 04:55:59.306207 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306139 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-token\") pod \"dafc8dfb-7244-4c01-93de-ec36a77d4918\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " Jul 21 04:55:59.306207 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306174 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-socket\") pod \"dafc8dfb-7244-4c01-93de-ec36a77d4918\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " Jul 21 04:55:59.306454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306218 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-data\") pod \"dafc8dfb-7244-4c01-93de-ec36a77d4918\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " Jul 21 04:55:59.306454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306257 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-credential-socket\") pod \"dafc8dfb-7244-4c01-93de-ec36a77d4918\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " Jul 21 04:55:59.306454 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306308 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hspk8\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-kube-api-access-hspk8\") pod \"dafc8dfb-7244-4c01-93de-ec36a77d4918\" (UID: \"dafc8dfb-7244-4c01-93de-ec36a77d4918\") " Jul 21 04:55:59.306617 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306442 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dafc8dfb-7244-4c01-93de-ec36a77d4918" (UID: "dafc8dfb-7244-4c01-93de-ec36a77d4918"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:55:59.306617 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306488 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dafc8dfb-7244-4c01-93de-ec36a77d4918" (UID: "dafc8dfb-7244-4c01-93de-ec36a77d4918"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:55:59.306617 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306560 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafc8dfb-7244-4c01-93de-ec36a77d4918-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dafc8dfb-7244-4c01-93de-ec36a77d4918" (UID: "dafc8dfb-7244-4c01-93de-ec36a77d4918"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:55:59.306775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306665 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dafc8dfb-7244-4c01-93de-ec36a77d4918" (UID: "dafc8dfb-7244-4c01-93de-ec36a77d4918"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:55:59.306841 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306822 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafc8dfb-7244-4c01-93de-ec36a77d4918-istiod-ca-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:59.306915 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306849 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:59.306915 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306863 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-workload-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:59.306915 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.306878 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-credential-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:59.307599 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.307567 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-data" (OuterVolumeSpecName: "istio-data") pod "dafc8dfb-7244-4c01-93de-ec36a77d4918" (UID: "dafc8dfb-7244-4c01-93de-ec36a77d4918"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:55:59.308644 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.308595 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-kube-api-access-hspk8" (OuterVolumeSpecName: "kube-api-access-hspk8") pod "dafc8dfb-7244-4c01-93de-ec36a77d4918" (UID: "dafc8dfb-7244-4c01-93de-ec36a77d4918"). InnerVolumeSpecName "kube-api-access-hspk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:55:59.308775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.308673 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-token" (OuterVolumeSpecName: "istio-token") pod "dafc8dfb-7244-4c01-93de-ec36a77d4918" (UID: "dafc8dfb-7244-4c01-93de-ec36a77d4918"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:55:59.308775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.308744 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dafc8dfb-7244-4c01-93de-ec36a77d4918" (UID: "dafc8dfb-7244-4c01-93de-ec36a77d4918"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 04:55:59.308897 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.308825 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dafc8dfb-7244-4c01-93de-ec36a77d4918" (UID: "dafc8dfb-7244-4c01-93de-ec36a77d4918"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:55:59.407285 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.407258 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hspk8\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-kube-api-access-hspk8\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:59.407285 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.407281 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-envoy\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:59.407473 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.407291 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-podinfo\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:59.407473 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.407298 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:59.407473 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.407306 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafc8dfb-7244-4c01-93de-ec36a77d4918-istio-data\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:55:59.979236 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.979204 2567 generic.go:356] "Generic (PLEG): container finished" podID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerID="15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc" exitCode=0 Jul 21 04:55:59.979432 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.979253 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" Jul 21 04:55:59.979432 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.979287 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" event={"ID":"dafc8dfb-7244-4c01-93de-ec36a77d4918","Type":"ContainerDied","Data":"15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc"} Jul 21 04:55:59.979432 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.979364 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47" event={"ID":"dafc8dfb-7244-4c01-93de-ec36a77d4918","Type":"ContainerDied","Data":"fb54e200e23045a9a77aafef59574dc26bd25b00ebe4c1baa0935daafaac411f"} Jul 21 04:55:59.979432 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.979397 2567 scope.go:117] "RemoveContainer" containerID="15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc" Jul 21 04:55:59.989903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.989887 2567 scope.go:117] "RemoveContainer" containerID="0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57" Jul 21 04:55:59.998149 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.998133 2567 scope.go:117] "RemoveContainer" containerID="15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc" Jul 21 04:55:59.998407 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:55:59.998388 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc\": container with ID starting with 15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc not found: ID does not exist" containerID="15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc" Jul 21 04:55:59.998477 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.998414 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc"} err="failed to get container status \"15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc\": rpc error: code = NotFound desc = could not find container \"15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc\": container with ID starting with 15eb650ac9376914da9fef7e524b80b1cd83a7d9c6fd53964db144bad7e5c8cc not found: ID does not exist" Jul 21 04:55:59.998477 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.998430 2567 scope.go:117] "RemoveContainer" containerID="0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57" Jul 21 04:55:59.998670 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:55:59.998655 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57\": container with ID starting with 0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57 not found: ID does not exist" containerID="0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57" Jul 21 04:55:59.998711 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:55:59.998673 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57"} err="failed to get container status \"0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57\": rpc error: code = NotFound desc = could not find container \"0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57\": container with ID starting with 0c2f812fb9f89a9bd31b4db371809840ade96d2a6706cd359854d8404559ad57 not found: ID does not exist" Jul 21 04:56:00.004228 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:00.004208 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47"] Jul 21 04:56:00.008711 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:00.008689 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-9fdb31-openshift-default-6d78b755c9-rbf47"] Jul 21 04:56:01.267266 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:01.267234 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" path="/var/lib/kubelet/pods/dafc8dfb-7244-4c01-93de-ec36a77d4918/volumes" Jul 21 04:56:07.969734 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:07.969701 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vd9b5"] Jul 21 04:56:16.358905 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:16.358875 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6"] Jul 21 04:56:16.359411 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:16.359118 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" containerID="cri-o://9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405" gracePeriod=30 Jul 21 04:56:21.617195 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.617167 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:56:21.701479 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701394 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-certs\") pod \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " Jul 21 04:56:21.701479 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701433 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-envoy\") pod \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " Jul 21 04:56:21.701479 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701468 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-podinfo\") pod \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " Jul 21 04:56:21.701762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701494 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-socket\") pod \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " Jul 21 04:56:21.701762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701543 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-credential-socket\") pod \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " Jul 21 04:56:21.701762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701675 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bdb2707a-dfcc-4712-8b10-9c395fef7b91" (UID: "bdb2707a-dfcc-4712-8b10-9c395fef7b91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:56:21.701762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701697 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l4sh\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-kube-api-access-2l4sh\") pod \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " Jul 21 04:56:21.701762 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701749 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-token\") pod \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " Jul 21 04:56:21.702023 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701781 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istiod-ca-cert\") pod \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " Jul 21 04:56:21.702023 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701822 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-data\") pod \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\" (UID: \"bdb2707a-dfcc-4712-8b10-9c395fef7b91\") " Jul 21 04:56:21.702023 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701817 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bdb2707a-dfcc-4712-8b10-9c395fef7b91" (UID: "bdb2707a-dfcc-4712-8b10-9c395fef7b91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:56:21.702023 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.701834 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bdb2707a-dfcc-4712-8b10-9c395fef7b91" (UID: "bdb2707a-dfcc-4712-8b10-9c395fef7b91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:56:21.702227 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.702128 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bdb2707a-dfcc-4712-8b10-9c395fef7b91" (UID: "bdb2707a-dfcc-4712-8b10-9c395fef7b91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 04:56:21.702227 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.702180 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-certs\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:56:21.702227 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.702199 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-workload-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:56:21.702227 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.702213 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-credential-socket\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:56:21.702758 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.702729 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-data" (OuterVolumeSpecName: "istio-data") pod "bdb2707a-dfcc-4712-8b10-9c395fef7b91" (UID: "bdb2707a-dfcc-4712-8b10-9c395fef7b91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:56:21.703903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.703870 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bdb2707a-dfcc-4712-8b10-9c395fef7b91" (UID: "bdb2707a-dfcc-4712-8b10-9c395fef7b91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 04:56:21.703903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.703898 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bdb2707a-dfcc-4712-8b10-9c395fef7b91" (UID: "bdb2707a-dfcc-4712-8b10-9c395fef7b91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 04:56:21.704095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.703930 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-kube-api-access-2l4sh" (OuterVolumeSpecName: "kube-api-access-2l4sh") pod "bdb2707a-dfcc-4712-8b10-9c395fef7b91" (UID: "bdb2707a-dfcc-4712-8b10-9c395fef7b91"). InnerVolumeSpecName "kube-api-access-2l4sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:56:21.704095 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.703943 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-token" (OuterVolumeSpecName: "istio-token") pod "bdb2707a-dfcc-4712-8b10-9c395fef7b91" (UID: "bdb2707a-dfcc-4712-8b10-9c395fef7b91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 04:56:21.803248 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.803223 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-podinfo\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:56:21.803248 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.803246 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l4sh\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-kube-api-access-2l4sh\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:56:21.803421 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.803256 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-token\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:56:21.803421 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.803264 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istiod-ca-cert\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:56:21.803421 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.803273 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-data\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:56:21.803421 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:21.803280 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdb2707a-dfcc-4712-8b10-9c395fef7b91-istio-envoy\") on node \"ip-10-0-137-226.ec2.internal\" DevicePath \"\"" Jul 21 04:56:22.117466 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.117435 2567 generic.go:356] "Generic (PLEG): container finished" podID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerID="9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405" exitCode=0 Jul 21 04:56:22.117642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.117498 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" Jul 21 04:56:22.117642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.117519 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" event={"ID":"bdb2707a-dfcc-4712-8b10-9c395fef7b91","Type":"ContainerDied","Data":"9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405"} Jul 21 04:56:22.117642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.117573 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6" event={"ID":"bdb2707a-dfcc-4712-8b10-9c395fef7b91","Type":"ContainerDied","Data":"5b18a090e39ce3944c0d1633dab414b5bfac032551c00de780549dd576de65ff"} Jul 21 04:56:22.117642 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.117596 2567 scope.go:117] "RemoveContainer" containerID="9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405" Jul 21 04:56:22.127468 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.127453 2567 scope.go:117] "RemoveContainer" containerID="4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e" Jul 21 04:56:22.135517 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.135499 2567 scope.go:117] "RemoveContainer" containerID="9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405" Jul 21 04:56:22.135792 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:56:22.135770 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405\": container with ID starting with 9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405 not found: ID does not exist" containerID="9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405" Jul 21 04:56:22.135880 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.135798 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405"} err="failed to get container status \"9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405\": rpc error: code = NotFound desc = could not find container \"9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405\": container with ID starting with 9ab3036fbe2010f755cb230f72ea772ba0031ee7c30f249cf3bf6747b2b52405 not found: ID does not exist" Jul 21 04:56:22.135880 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.135816 2567 scope.go:117] "RemoveContainer" containerID="4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e" Jul 21 04:56:22.136058 ip-10-0-137-226 kubenswrapper[2567]: E0721 04:56:22.136038 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e\": container with ID starting with 4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e not found: ID does not exist" containerID="4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e" Jul 21 04:56:22.136124 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.136063 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e"} err="failed to get container status \"4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e\": rpc error: code = NotFound desc = could not find container \"4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e\": container with ID starting with 4bdd21e7380397e28b2f87e92d7010cbba298539ceed5c9f596f6c0b3be5db7e not found: ID does not exist" Jul 21 04:56:22.143465 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.143444 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6"] Jul 21 04:56:22.147122 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.147103 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-4de716-openshift-default-78d78cfd75-4x6l6"] Jul 21 04:56:22.714632 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.714604 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cv4fn_fbf45a96-3933-44e2-9eaa-df1310cf26b1/manager/0.log" Jul 21 04:56:22.840809 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:22.840776 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jtzss_eba1f514-8403-4696-8c2d-cc2670d4f072/manager/0.log" Jul 21 04:56:23.208128 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:23.208095 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-l8l94_c8a817cb-3511-4266-980f-7e1a5ce1c10b/manager/0.log" Jul 21 04:56:23.266586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:23.266557 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" path="/var/lib/kubelet/pods/bdb2707a-dfcc-4712-8b10-9c395fef7b91/volumes" Jul 21 04:56:23.576535 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:23.576504 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d877cbcc8-vpqz6_4f12d8d5-ea89-4a60-86d1-3cf819c15488/manager/0.log" Jul 21 04:56:23.583649 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:23.583628 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d877cbcc8-vpqz6_4f12d8d5-ea89-4a60-86d1-3cf819c15488/copy-manifests/0.log" Jul 21 04:56:24.280033 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:24.280006 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-xrgs8_bfa53375-310e-453d-85b0-a01f06ee944c/postgres/0.log" Jul 21 04:56:25.107735 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.107695 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq_e67cda2d-9360-469e-b915-0d3b3568e60f/util/0.log" Jul 21 04:56:25.114479 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.114451 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq_e67cda2d-9360-469e-b915-0d3b3568e60f/pull/0.log" Jul 21 04:56:25.120780 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.120760 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq_e67cda2d-9360-469e-b915-0d3b3568e60f/extract/0.log" Jul 21 04:56:25.235688 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.235660 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4_a1dac966-414e-4aee-9d1c-efe3c5c80e81/util/0.log" Jul 21 04:56:25.242813 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.242791 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4_a1dac966-414e-4aee-9d1c-efe3c5c80e81/pull/0.log" Jul 21 04:56:25.249688 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.249672 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4_a1dac966-414e-4aee-9d1c-efe3c5c80e81/extract/0.log" Jul 21 04:56:25.362218 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.362159 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb_c785cd48-ef6a-476b-9e4d-ea8a4bf994cb/util/0.log" Jul 21 04:56:25.369599 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.369582 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb_c785cd48-ef6a-476b-9e4d-ea8a4bf994cb/pull/0.log" Jul 21 04:56:25.376862 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.376838 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb_c785cd48-ef6a-476b-9e4d-ea8a4bf994cb/extract/0.log" Jul 21 04:56:25.500043 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.500018 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl_3b0dffcc-329b-45ff-be93-41b3b52eff4a/util/0.log" Jul 21 04:56:25.507136 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.507117 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl_3b0dffcc-329b-45ff-be93-41b3b52eff4a/pull/0.log" Jul 21 04:56:25.514303 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.514288 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl_3b0dffcc-329b-45ff-be93-41b3b52eff4a/extract/0.log" Jul 21 04:56:25.764276 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.764251 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2zzxt_61a06832-b305-4be3-95b9-78e351681fa0/manager/0.log" Jul 21 04:56:25.898797 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:25.898774 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rtsvd_3ecc4112-39f8-4744-9967-8c73df953f92/manager/0.log" Jul 21 04:56:26.027026 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:26.026957 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-88r7n_248149d4-7aed-4d5f-98e0-102719e9f638/kuadrant-console-plugin/0.log" Jul 21 04:56:26.153961 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:26.153933 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-576mf_0c9ba075-91dd-440e-ba17-c235f89b3e9f/registry-server/0.log" Jul 21 04:56:26.286325 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:26.286260 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2zxbd_3f418020-20c1-496e-b74e-03845819655d/manager/0.log" Jul 21 04:56:26.409993 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:26.409971 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vd9b5_f93e01da-f748-4c81-9234-812c9acc8772/limitador/0.log" Jul 21 04:56:27.265702 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:27.265673 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-757f6cbd6c-82drq_3bdb37ce-942f-4d36-94e4-b09abc0e200b/kube-auth-proxy/0.log" Jul 21 04:56:27.388655 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:27.388617 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rdt7k_0218007d-5768-4855-ac24-f35d5daf2d66/istio-proxy/1.log" Jul 21 04:56:27.754247 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:27.754175 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6fb748b6-ncql2_ded6b4a0-3715-4e78-8bce-7505e977b0aa/router/0.log" Jul 21 04:56:28.100120 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:28.100097 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nh96q_ae8b8ed2-a6d2-4a72-99bd-b41fb555f3e6/main/0.log" Jul 21 04:56:28.346752 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:28.346724 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-s6nsk_ffce86ca-f434-4191-ac65-1da3b88b22c7/main/0.log" Jul 21 04:56:34.784285 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784249 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6wgn4/must-gather-2h8j5"] Jul 21 04:56:34.784728 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784713 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" Jul 21 04:56:34.784775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784730 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" Jul 21 04:56:34.784775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784744 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd32518d-8a29-4151-9229-a159ab19fad0" containerName="test-curl-7799-140183462626768" Jul 21 04:56:34.784775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784751 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd32518d-8a29-4151-9229-a159ab19fad0" containerName="test-curl-7799-140183462626768" Jul 21 04:56:34.784775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784764 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" Jul 21 04:56:34.784775 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784773 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" Jul 21 04:56:34.784931 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784788 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" Jul 21 04:56:34.784931 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784794 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" Jul 21 04:56:34.784931 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784867 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" Jul 21 04:56:34.784931 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784877 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" Jul 21 04:56:34.784931 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784883 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" Jul 21 04:56:34.784931 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784889 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd32518d-8a29-4151-9229-a159ab19fad0" containerName="test-curl-7799-140183462626768" Jul 21 04:56:34.785105 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784957 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" Jul 21 04:56:34.785105 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.784964 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dafc8dfb-7244-4c01-93de-ec36a77d4918" containerName="istio-proxy" Jul 21 04:56:34.785105 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.785027 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb2707a-dfcc-4712-8b10-9c395fef7b91" containerName="istio-proxy" Jul 21 04:56:34.788136 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.788116 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6wgn4/must-gather-2h8j5" Jul 21 04:56:34.790557 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.790537 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6wgn4\"/\"default-dockercfg-kztqk\"" Jul 21 04:56:34.790741 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.790728 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6wgn4\"/\"openshift-service-ca.crt\"" Jul 21 04:56:34.791593 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.791580 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6wgn4\"/\"kube-root-ca.crt\"" Jul 21 04:56:34.797469 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.797443 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6wgn4/must-gather-2h8j5"] Jul 21 04:56:34.912828 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.912792 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d8a7300-78ea-45e5-858a-cb922428fcd7-must-gather-output\") pod \"must-gather-2h8j5\" (UID: \"5d8a7300-78ea-45e5-858a-cb922428fcd7\") " pod="openshift-must-gather-6wgn4/must-gather-2h8j5" Jul 21 04:56:34.912956 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:34.912867 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2b7\" (UniqueName: \"kubernetes.io/projected/5d8a7300-78ea-45e5-858a-cb922428fcd7-kube-api-access-jj2b7\") pod \"must-gather-2h8j5\" (UID: \"5d8a7300-78ea-45e5-858a-cb922428fcd7\") " pod="openshift-must-gather-6wgn4/must-gather-2h8j5" Jul 21 04:56:35.013589 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:35.013555 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2b7\" (UniqueName: \"kubernetes.io/projected/5d8a7300-78ea-45e5-858a-cb922428fcd7-kube-api-access-jj2b7\") pod \"must-gather-2h8j5\" (UID: \"5d8a7300-78ea-45e5-858a-cb922428fcd7\") " pod="openshift-must-gather-6wgn4/must-gather-2h8j5" Jul 21 04:56:35.013750 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:35.013654 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d8a7300-78ea-45e5-858a-cb922428fcd7-must-gather-output\") pod \"must-gather-2h8j5\" (UID: \"5d8a7300-78ea-45e5-858a-cb922428fcd7\") " pod="openshift-must-gather-6wgn4/must-gather-2h8j5" Jul 21 04:56:35.013985 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:35.013965 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5d8a7300-78ea-45e5-858a-cb922428fcd7-must-gather-output\") pod \"must-gather-2h8j5\" (UID: \"5d8a7300-78ea-45e5-858a-cb922428fcd7\") " pod="openshift-must-gather-6wgn4/must-gather-2h8j5" Jul 21 04:56:35.025078 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:35.025050 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2b7\" (UniqueName: \"kubernetes.io/projected/5d8a7300-78ea-45e5-858a-cb922428fcd7-kube-api-access-jj2b7\") pod \"must-gather-2h8j5\" (UID: \"5d8a7300-78ea-45e5-858a-cb922428fcd7\") " pod="openshift-must-gather-6wgn4/must-gather-2h8j5" Jul 21 04:56:35.098751 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:35.098694 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6wgn4/must-gather-2h8j5" Jul 21 04:56:35.230960 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:35.230933 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6wgn4/must-gather-2h8j5"] Jul 21 04:56:35.232460 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:56:35.232434 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d8a7300_78ea_45e5_858a_cb922428fcd7.slice/crio-af8efe3681e636ada939167acfe9098487cc8d430d52f6feb906acf0813a11e0 WatchSource:0}: Error finding container af8efe3681e636ada939167acfe9098487cc8d430d52f6feb906acf0813a11e0: Status 404 returned error can't find the container with id af8efe3681e636ada939167acfe9098487cc8d430d52f6feb906acf0813a11e0 Jul 21 04:56:36.199739 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:36.199650 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6wgn4/must-gather-2h8j5" event={"ID":"5d8a7300-78ea-45e5-858a-cb922428fcd7","Type":"ContainerStarted","Data":"9440236cfc34046b9c18ebfe55b4d4ce3dbbc1ab4920443edd4ba014a0e1e5b1"} Jul 21 04:56:36.199739 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:36.199704 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6wgn4/must-gather-2h8j5" event={"ID":"5d8a7300-78ea-45e5-858a-cb922428fcd7","Type":"ContainerStarted","Data":"fbd5f9e06145b3fd530e5e1db277917f7270888aa4c0a6fcf9f0617c02d12045"} Jul 21 04:56:36.199739 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:36.199724 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6wgn4/must-gather-2h8j5" event={"ID":"5d8a7300-78ea-45e5-858a-cb922428fcd7","Type":"ContainerStarted","Data":"af8efe3681e636ada939167acfe9098487cc8d430d52f6feb906acf0813a11e0"} Jul 21 04:56:36.216864 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:36.216812 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6wgn4/must-gather-2h8j5" podStartSLOduration=1.535187662 podStartE2EDuration="2.21679256s" totalImagesPullingTime="681.604898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 04:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:56:36.215068694 +0000 UTC m=+2841.614401451" watchObservedRunningTime="2026-07-21 04:56:36.21679256 +0000 UTC m=+2841.616125319" Jul 21 04:56:37.554152 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:37.554118 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wn4dg_c21ab5db-d116-4b5f-8f61-df4fe3a0ef89/global-pull-secret-syncer/0.log" Jul 21 04:56:37.636215 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:37.636182 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j676g_6254d1d7-216d-4417-9fcb-6f71dc38129b/konnectivity-agent/0.log" Jul 21 04:56:37.709489 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:37.709415 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-226.ec2.internal_3e07ae959d76e4e217922a691b6e844d/haproxy/0.log" Jul 21 04:56:42.046957 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.046921 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq_e67cda2d-9360-469e-b915-0d3b3568e60f/extract/0.log" Jul 21 04:56:42.081743 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.081711 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq_e67cda2d-9360-469e-b915-0d3b3568e60f/util/0.log" Jul 21 04:56:42.110219 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.110179 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xz4qq_e67cda2d-9360-469e-b915-0d3b3568e60f/pull/0.log" Jul 21 04:56:42.149452 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.149425 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4_a1dac966-414e-4aee-9d1c-efe3c5c80e81/extract/0.log" Jul 21 04:56:42.172392 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.172364 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4_a1dac966-414e-4aee-9d1c-efe3c5c80e81/util/0.log" Jul 21 04:56:42.198430 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.198395 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0njdm4_a1dac966-414e-4aee-9d1c-efe3c5c80e81/pull/0.log" Jul 21 04:56:42.227131 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.227100 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb_c785cd48-ef6a-476b-9e4d-ea8a4bf994cb/extract/0.log" Jul 21 04:56:42.262720 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.262630 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb_c785cd48-ef6a-476b-9e4d-ea8a4bf994cb/util/0.log" Jul 21 04:56:42.294231 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.294188 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73nwnzb_c785cd48-ef6a-476b-9e4d-ea8a4bf994cb/pull/0.log" Jul 21 04:56:42.340484 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.340401 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl_3b0dffcc-329b-45ff-be93-41b3b52eff4a/extract/0.log" Jul 21 04:56:42.364513 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.364477 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl_3b0dffcc-329b-45ff-be93-41b3b52eff4a/util/0.log" Jul 21 04:56:42.388933 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.388893 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jl4xl_3b0dffcc-329b-45ff-be93-41b3b52eff4a/pull/0.log" Jul 21 04:56:42.752917 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.752846 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2zzxt_61a06832-b305-4be3-95b9-78e351681fa0/manager/0.log" Jul 21 04:56:42.776383 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.776354 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-rtsvd_3ecc4112-39f8-4744-9967-8c73df953f92/manager/0.log" Jul 21 04:56:42.803059 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.803033 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-88r7n_248149d4-7aed-4d5f-98e0-102719e9f638/kuadrant-console-plugin/0.log" Jul 21 04:56:42.851586 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:42.851550 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-576mf_0c9ba075-91dd-440e-ba17-c235f89b3e9f/registry-server/0.log" Jul 21 04:56:43.023246 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:43.023215 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-2zxbd_3f418020-20c1-496e-b74e-03845819655d/manager/0.log" Jul 21 04:56:43.055002 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:43.054972 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vd9b5_f93e01da-f748-4c81-9234-812c9acc8772/limitador/0.log" Jul 21 04:56:44.970946 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:44.970855 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-zzs2c_6889f625-36b3-431e-9b82-c71e379c1cee/cluster-monitoring-operator/0.log" Jul 21 04:56:45.313817 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:45.313783 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-whl8l_6b2a90b8-0f3b-4be6-93b3-b6de043464e5/node-exporter/0.log" Jul 21 04:56:45.337779 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:45.337745 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-whl8l_6b2a90b8-0f3b-4be6-93b3-b6de043464e5/kube-rbac-proxy/0.log" Jul 21 04:56:45.360010 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:45.359980 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-whl8l_6b2a90b8-0f3b-4be6-93b3-b6de043464e5/init-textfile/0.log" Jul 21 04:56:47.076175 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.076140 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-6b4vp_70bac8e0-64e1-437e-878e-d67cf04577ce/networking-console-plugin/0.log" Jul 21 04:56:47.138533 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.138498 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g"] Jul 21 04:56:47.146168 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.146130 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.150969 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.150939 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g"] Jul 21 04:56:47.242109 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.242028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfsl\" (UniqueName: \"kubernetes.io/projected/b2799044-015e-40da-87f2-2e12e520c950-kube-api-access-kkfsl\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.242418 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.242396 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-podres\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.242702 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.242680 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-proc\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.242934 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.242916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-lib-modules\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.243897 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.243101 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-sys\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.345376 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.345282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-sys\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.345376 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.345371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfsl\" (UniqueName: \"kubernetes.io/projected/b2799044-015e-40da-87f2-2e12e520c950-kube-api-access-kkfsl\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.345622 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.345397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-podres\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.345622 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.345433 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-sys\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.345622 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.345455 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-proc\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.345622 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.345515 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-proc\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.345622 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.345573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-lib-modules\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.345622 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.345579 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-podres\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.345881 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.345703 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2799044-015e-40da-87f2-2e12e520c950-lib-modules\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.360734 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.360701 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfsl\" (UniqueName: \"kubernetes.io/projected/b2799044-015e-40da-87f2-2e12e520c950-kube-api-access-kkfsl\") pod \"perf-node-gather-daemonset-62g6g\" (UID: \"b2799044-015e-40da-87f2-2e12e520c950\") " pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.461620 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.461579 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:47.650903 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.650538 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g"] Jul 21 04:56:47.652421 ip-10-0-137-226 kubenswrapper[2567]: W0721 04:56:47.652385 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb2799044_015e_40da_87f2_2e12e520c950.slice/crio-644ac722d9bb0e7b6d1970225998ebdb6e52d87cf26e0eae6d99abac5a701d2f WatchSource:0}: Error finding container 644ac722d9bb0e7b6d1970225998ebdb6e52d87cf26e0eae6d99abac5a701d2f: Status 404 returned error can't find the container with id 644ac722d9bb0e7b6d1970225998ebdb6e52d87cf26e0eae6d99abac5a701d2f Jul 21 04:56:47.777138 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.777110 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/2.log" Jul 21 04:56:47.783023 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:47.783004 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-g9hm8_97374277-5b6f-4196-b591-b152148b4b33/console-operator/3.log" Jul 21 04:56:48.281503 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:48.281470 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-964f5b447-8hjqx_e6412c0e-d008-46d9-9296-cfdc4ebe666e/console/0.log" Jul 21 04:56:48.304576 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:48.304546 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" event={"ID":"b2799044-015e-40da-87f2-2e12e520c950","Type":"ContainerStarted","Data":"5a1932d2ca38ebd5aa36f8b25a2a73d5e3b175ebb193a2393652fc11940d281c"} Jul 21 04:56:48.304791 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:48.304637 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" event={"ID":"b2799044-015e-40da-87f2-2e12e520c950","Type":"ContainerStarted","Data":"644ac722d9bb0e7b6d1970225998ebdb6e52d87cf26e0eae6d99abac5a701d2f"} Jul 21 04:56:48.304791 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:48.304703 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:48.322198 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:48.322140 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" podStartSLOduration=1.322121716 podStartE2EDuration="1.322121716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 04:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 04:56:48.32055267 +0000 UTC m=+2853.719885419" watchObservedRunningTime="2026-07-21 04:56:48.322121716 +0000 UTC m=+2853.721454474" Jul 21 04:56:48.955680 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:48.955646 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-4nc27_d05da316-6261-4ce7-9c1e-53ffa7ff58bc/volume-data-source-validator/0.log" Jul 21 04:56:50.044458 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:50.044423 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wx6xf_564513d6-3759-4aa0-9318-bc34a30a6bbb/dns/0.log" Jul 21 04:56:50.064692 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:50.064642 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wx6xf_564513d6-3759-4aa0-9318-bc34a30a6bbb/kube-rbac-proxy/0.log" Jul 21 04:56:50.111113 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:50.111090 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bkscj_406658cc-5989-4a5e-8ef8-a27780de41a5/dns-node-resolver/0.log" Jul 21 04:56:50.699141 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:50.699109 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-snqhr_5ab7d0e5-db4f-4988-80a9-4edbd26c174c/node-ca/0.log" Jul 21 04:56:51.840992 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:51.840955 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-757f6cbd6c-82drq_3bdb37ce-942f-4d36-94e4-b09abc0e200b/kube-auth-proxy/0.log" Jul 21 04:56:51.891636 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:51.891595 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rdt7k_0218007d-5768-4855-ac24-f35d5daf2d66/istio-proxy/0.log" Jul 21 04:56:51.925431 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:51.925394 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rdt7k_0218007d-5768-4855-ac24-f35d5daf2d66/istio-proxy/1.log" Jul 21 04:56:52.046941 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:52.046914 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6fb748b6-ncql2_ded6b4a0-3715-4e78-8bce-7505e977b0aa/router/0.log" Jul 21 04:56:52.805124 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:52.805095 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hwcbj_8f8543d9-2530-4536-b660-85e318bf3d60/serve-healthcheck-canary/0.log" Jul 21 04:56:53.274878 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:53.274847 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-7r77f_19d55741-8411-4b6b-91fd-599aeb651f83/insights-operator/0.log" Jul 21 04:56:53.277522 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:53.277496 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-7r77f_19d55741-8411-4b6b-91fd-599aeb651f83/insights-operator/1.log" Jul 21 04:56:53.367072 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:53.367045 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lfdqt_01a716db-4571-4add-99cb-fb4c3ceff522/kube-rbac-proxy/0.log" Jul 21 04:56:53.389186 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:53.389159 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lfdqt_01a716db-4571-4add-99cb-fb4c3ceff522/exporter/0.log" Jul 21 04:56:53.411056 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:53.411028 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lfdqt_01a716db-4571-4add-99cb-fb4c3ceff522/extractor/0.log" Jul 21 04:56:54.323030 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:54.323003 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6wgn4/perf-node-gather-daemonset-62g6g" Jul 21 04:56:55.531028 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:55.530994 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cv4fn_fbf45a96-3933-44e2-9eaa-df1310cf26b1/manager/0.log" Jul 21 04:56:55.783135 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:55.783060 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-jtzss_eba1f514-8403-4696-8c2d-cc2670d4f072/manager/0.log" Jul 21 04:56:56.021056 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:56.021023 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-l8l94_c8a817cb-3511-4266-980f-7e1a5ce1c10b/manager/0.log" Jul 21 04:56:56.202528 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:56.202495 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d877cbcc8-vpqz6_4f12d8d5-ea89-4a60-86d1-3cf819c15488/manager/0.log" Jul 21 04:56:56.228157 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:56.228131 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d877cbcc8-vpqz6_4f12d8d5-ea89-4a60-86d1-3cf819c15488/copy-manifests/0.log" Jul 21 04:56:57.623260 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:57.623227 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-757f6cbd6c-82drq_3bdb37ce-942f-4d36-94e4-b09abc0e200b/kube-auth-proxy/0.log" Jul 21 04:56:57.642300 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:57.642277 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rdt7k_0218007d-5768-4855-ac24-f35d5daf2d66/istio-proxy/0.log" Jul 21 04:56:57.675793 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:57.675764 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rdt7k_0218007d-5768-4855-ac24-f35d5daf2d66/istio-proxy/1.log" Jul 21 04:56:57.739286 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:57.739256 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6fb748b6-ncql2_ded6b4a0-3715-4e78-8bce-7505e977b0aa/router/0.log" Jul 21 04:56:59.137621 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:59.137549 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-757f6cbd6c-82drq_3bdb37ce-942f-4d36-94e4-b09abc0e200b/kube-auth-proxy/0.log" Jul 21 04:56:59.159378 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:59.158231 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rdt7k_0218007d-5768-4855-ac24-f35d5daf2d66/istio-proxy/0.log" Jul 21 04:56:59.193164 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:59.193140 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rdt7k_0218007d-5768-4855-ac24-f35d5daf2d66/istio-proxy/1.log" Jul 21 04:56:59.260957 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:56:59.260931 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c6fb748b6-ncql2_ded6b4a0-3715-4e78-8bce-7505e977b0aa/router/0.log" Jul 21 04:57:02.842749 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:02.842714 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-zntm6_4297ba8e-5cf6-40a5-90ba-2d1215b9237f/kube-storage-version-migrator-operator/1.log" Jul 21 04:57:02.843760 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:02.843733 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-zntm6_4297ba8e-5cf6-40a5-90ba-2d1215b9237f/kube-storage-version-migrator-operator/0.log" Jul 21 04:57:03.861220 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:03.861190 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-897jc_af409d50-2522-42a5-bada-8126ed1944c3/kube-multus/0.log" Jul 21 04:57:03.885707 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:03.885678 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtmdb_3e5da763-6747-4693-af09-9370b50f6e56/kube-multus-additional-cni-plugins/0.log" Jul 21 04:57:03.907117 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:03.907095 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtmdb_3e5da763-6747-4693-af09-9370b50f6e56/egress-router-binary-copy/0.log" Jul 21 04:57:03.928416 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:03.928394 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtmdb_3e5da763-6747-4693-af09-9370b50f6e56/cni-plugins/0.log" Jul 21 04:57:03.950377 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:03.950356 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtmdb_3e5da763-6747-4693-af09-9370b50f6e56/bond-cni-plugin/0.log" Jul 21 04:57:03.971712 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:03.971683 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtmdb_3e5da763-6747-4693-af09-9370b50f6e56/routeoverride-cni/0.log" Jul 21 04:57:03.994989 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:03.994965 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtmdb_3e5da763-6747-4693-af09-9370b50f6e56/whereabouts-cni-bincopy/0.log" Jul 21 04:57:04.020713 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:04.020691 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtmdb_3e5da763-6747-4693-af09-9370b50f6e56/whereabouts-cni/0.log" Jul 21 04:57:04.429589 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:04.429557 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8w86n_fabd6623-e601-4828-8b83-673db7105de6/network-metrics-daemon/0.log" Jul 21 04:57:04.450500 ip-10-0-137-226 kubenswrapper[2567]: I0721 04:57:04.450469 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8w86n_fabd6623-e601-4828-8b83-673db7105de6/kube-rbac-proxy/0.log"