Jul 21 20:11:59.444241 ip-10-0-141-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 20:11:59.444252 ip-10-0-141-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 20:11:59.444260 ip-10-0-141-132 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 20:11:59.444542 ip-10-0-141-132 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 20:12:09.474179 ip-10-0-141-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 20:12:09.474195 ip-10-0-141-132 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e5588f76ccdb4edc9440d2ef6105e512 -- Jul 21 20:14:05.217339 ip-10-0-141-132 systemd[1]: Starting Kubernetes Kubelet... Jul 21 20:14:05.558333 ip-10-0-141-132 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 20:14:05.558333 ip-10-0-141-132 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 20:14:05.558333 ip-10-0-141-132 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 20:14:05.558333 ip-10-0-141-132 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 20:14:05.558333 ip-10-0-141-132 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 20:14:05.560460 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.560371 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 20:14:05.565731 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565700 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 20:14:05.565731 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565726 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 20:14:05.565731 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565731 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 20:14:05.565731 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565734 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 20:14:05.565731 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565739 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565744 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565748 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565751 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565754 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565757 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565760 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565763 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565766 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565771 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565774 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565778 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565780 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565783 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565785 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565788 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565791 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565793 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565796 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 20:14:05.565962 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565799 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565802 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565804 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565807 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565816 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565819 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565821 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565824 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565827 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565829 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565832 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565834 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565838 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565840 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565843 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565847 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565850 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565852 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565855 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565858 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 20:14:05.566408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565860 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565863 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565865 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565868 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565870 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565873 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565875 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565878 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565880 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565882 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565885 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565888 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565890 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565893 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565896 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565899 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565902 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565904 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565907 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565909 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 20:14:05.566884 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565912 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565914 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565917 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565919 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565922 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565925 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565927 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565930 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565932 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565935 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565955 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565959 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565962 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565965 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565968 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565971 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565973 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565976 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565979 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565981 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 20:14:05.567375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565985 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565989 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565991 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565994 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565996 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.565999 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566002 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566004 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566007 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566010 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566012 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566015 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566019 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566023 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566026 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566029 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566758 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566765 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566769 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 20:14:05.567840 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566772 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566777 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566781 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566784 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566787 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566790 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566793 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566795 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566798 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566800 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566804 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566808 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566811 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566813 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566816 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566819 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566821 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566824 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566827 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 20:14:05.568312 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566829 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566832 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566834 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566837 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566840 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566842 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566845 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566848 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566850 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566854 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566856 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566859 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566862 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566864 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566867 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566869 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566872 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566874 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566877 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566879 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 20:14:05.568776 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566882 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566884 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566887 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566890 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566892 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566895 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566898 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566900 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566904 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566908 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566912 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566914 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566917 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566919 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566922 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566924 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566927 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566929 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566932 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566934 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 20:14:05.569262 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566971 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566976 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566979 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566983 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566986 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566989 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566992 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566995 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.566997 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567000 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567003 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567005 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567008 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567010 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567013 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567015 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567018 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567020 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567023 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567025 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 20:14:05.569746 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567027 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567031 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567034 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567037 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567040 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567043 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567045 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567048 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567050 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567053 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567055 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567058 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567061 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567063 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567065 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567068 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567071 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567515 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567521 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 20:14:05.570245 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567525 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567528 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567531 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567534 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567537 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567540 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567543 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567546 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567549 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567551 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567554 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567556 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567559 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567561 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567565 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567568 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567571 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567574 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567577 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567579 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 20:14:05.570703 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567582 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567585 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567588 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567591 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567593 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567595 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567598 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567601 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567603 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567606 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567608 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567611 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567613 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567616 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567618 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567621 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567623 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567626 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567629 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 20:14:05.571204 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567633 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567637 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567640 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567643 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567647 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567650 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567654 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567657 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567660 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567663 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567666 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567668 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567671 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567673 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567678 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567681 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567684 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567687 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567689 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 20:14:05.571711 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567693 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567695 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567698 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567700 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567702 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567705 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567707 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567710 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567712 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567715 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567717 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567720 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567722 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567725 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567727 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567730 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567733 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567735 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567738 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567740 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 20:14:05.572172 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567745 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567748 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567751 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567753 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567756 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567758 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567761 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567763 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567766 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567768 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567771 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567773 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567776 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567778 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567781 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567783 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567786 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567788 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.567791 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567866 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567881 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 20:14:05.572640 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567890 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567895 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567899 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567903 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567909 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567913 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567917 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567920 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567924 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567927 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567930 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567934 2567 flags.go:64] FLAG: --cgroup-root="" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567953 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567957 2567 flags.go:64] FLAG: --client-ca-file="" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567961 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567964 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567968 2567 flags.go:64] FLAG: --cluster-domain="" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567971 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567974 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567977 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567980 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567985 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567988 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 20:14:05.573187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567991 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567995 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.567998 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568000 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568004 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568007 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568010 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568014 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568017 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568020 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568023 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568026 2567 flags.go:64] FLAG: --enable-server="true" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568029 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568033 2567 flags.go:64] FLAG: --event-burst="100" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568036 2567 flags.go:64] FLAG: --event-qps="50" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568039 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568042 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568045 2567 flags.go:64] FLAG: --eviction-hard="" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568049 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568052 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568056 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568059 2567 flags.go:64] FLAG: --eviction-soft="" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568062 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568065 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568068 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 20:14:05.573727 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568071 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568074 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568077 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568079 2567 flags.go:64] FLAG: --feature-gates="" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568084 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568087 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568090 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568093 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568096 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568099 2567 flags.go:64] FLAG: --help="false" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568101 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-141-132.ec2.internal" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568105 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568108 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568110 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568114 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568117 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568120 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568123 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568126 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568129 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568132 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568135 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568138 2567 flags.go:64] FLAG: --kube-reserved="" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568140 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 20:14:05.574367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568143 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568146 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568149 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568153 2567 flags.go:64] FLAG: --lock-file="" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568155 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568158 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568161 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568167 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568170 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568173 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568176 2567 flags.go:64] FLAG: --logging-format="text" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568178 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568182 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568184 2567 flags.go:64] FLAG: --manifest-url="" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568189 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568193 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568196 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568200 2567 flags.go:64] FLAG: --max-pods="110" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568203 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568206 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568209 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568212 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568215 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568218 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568221 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 20:14:05.575169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568229 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568232 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568235 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568238 2567 flags.go:64] FLAG: --pod-cidr="" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568241 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568246 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568249 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568252 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568255 2567 flags.go:64] FLAG: --port="10250" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568258 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568261 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d131d4770b9b7a3" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568265 2567 flags.go:64] FLAG: --qos-reserved="" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568268 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568271 2567 flags.go:64] FLAG: --register-node="true" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568274 2567 flags.go:64] FLAG: --register-with-taints="" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568278 2567 flags.go:64] FLAG: --registry-burst="10" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568281 2567 flags.go:64] FLAG: --registry-qps="5" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568284 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568287 2567 flags.go:64] FLAG: --reserved-memory="" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568290 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568294 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568299 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568302 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568305 2567 flags.go:64] FLAG: --runonce="false" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568308 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 20:14:05.576246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568311 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568314 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568318 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568321 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568324 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568327 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568330 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568333 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568336 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568339 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568341 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568345 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568347 2567 flags.go:64] FLAG: --system-cgroups="" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568350 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568356 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568358 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568361 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568365 2567 flags.go:64] FLAG: --tls-min-version="" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568369 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568371 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568374 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568377 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568380 2567 flags.go:64] FLAG: --v="2" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568384 2567 flags.go:64] FLAG: --version="false" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568389 2567 flags.go:64] FLAG: --vmodule="" Jul 21 20:14:05.577302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568393 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.568396 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568520 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568526 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568529 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568532 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568535 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568538 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568541 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568544 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568546 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568549 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568552 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568555 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568558 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568560 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568563 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568565 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568568 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568570 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 20:14:05.578456 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568573 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568575 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568578 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568581 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568583 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568587 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568590 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568593 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568595 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568598 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568600 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568603 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568606 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568609 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568611 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568615 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568618 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568620 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568623 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568625 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 20:14:05.579149 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568628 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568630 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568633 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568636 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568638 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568641 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568644 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568646 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568649 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568652 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568654 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568656 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568660 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568663 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568666 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568669 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568671 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568675 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568678 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 20:14:05.579863 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568681 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568683 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568686 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568688 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568691 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568694 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568697 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568699 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568703 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568706 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568709 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568713 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568716 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568719 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568723 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568726 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568729 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568732 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568734 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 20:14:05.580667 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568737 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568740 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568743 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568746 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568749 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568752 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568754 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568757 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568759 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568762 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568764 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568768 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568771 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568773 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568776 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568779 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568781 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568784 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568787 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568790 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 20:14:05.581247 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568793 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568796 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.568799 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.569508 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.577248 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.577271 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.577298 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.577307 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577396 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577405 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577410 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577415 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577420 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577425 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577429 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 20:14:05.581865 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577434 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577439 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577443 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577447 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577451 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577456 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577460 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577465 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577469 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577474 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577480 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577485 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577489 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577494 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577499 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577503 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577507 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577513 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577518 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577523 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 20:14:05.582298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577527 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577532 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577537 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577542 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577546 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577550 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577554 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577558 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577562 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577567 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577571 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577575 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577580 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577584 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577588 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577592 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577596 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577601 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577605 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577609 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 20:14:05.582794 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577613 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577617 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577622 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577626 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577630 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577635 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577639 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577643 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577650 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577656 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577662 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577667 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577672 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577678 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577683 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577688 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577693 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577697 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577701 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 20:14:05.583340 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577707 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577713 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577718 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577722 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577727 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577731 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577735 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577740 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577744 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577748 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577753 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577757 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577761 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577765 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577769 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577773 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577778 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577782 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577787 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 20:14:05.583864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577791 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577795 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577799 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577804 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577808 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577813 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577817 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577822 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577827 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577832 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577838 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577843 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577848 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.577853 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.577862 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 20:14:05.584431 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578065 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578075 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578080 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578084 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578089 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578093 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578101 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578107 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578112 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578116 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578120 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578125 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578129 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578133 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578137 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578142 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578147 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578151 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578155 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578160 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 20:14:05.584815 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578164 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578169 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578173 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578178 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578182 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578187 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578192 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578197 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578204 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578208 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578213 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578217 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578221 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578225 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578230 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578234 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578238 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578242 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578247 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578252 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 20:14:05.585412 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578256 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578260 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578264 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578269 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578273 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578277 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578281 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578286 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578290 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578294 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578298 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578303 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578308 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578312 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578316 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578320 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578325 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578330 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578334 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 20:14:05.585901 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578338 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578342 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578346 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578351 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578355 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578359 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578363 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578367 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578372 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578376 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578380 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578384 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578388 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578392 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578396 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578401 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578405 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578409 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578413 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578417 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 20:14:05.586658 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578422 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578426 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578430 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578434 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578439 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578443 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578447 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578451 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578455 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578458 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578464 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578467 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578472 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578475 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578479 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578484 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578488 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578492 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578496 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 20:14:05.587479 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:05.578503 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 20:14:05.588437 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.578512 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 20:14:05.588437 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.578699 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 20:14:05.588437 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.581436 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 20:14:05.588437 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.582785 2567 server.go:1019] "Starting client certificate rotation" Jul 21 20:14:05.588437 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.582893 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 20:14:05.588437 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.582934 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 20:14:05.588437 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.586568 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 20:14:05.590998 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.590971 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 20:14:05.598193 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.598169 2567 log.go:25] "Validated CRI v1 runtime API" Jul 21 20:14:05.600389 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.600371 2567 log.go:25] "Validated CRI v1 image API" Jul 21 20:14:05.601168 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.601152 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 20:14:05.603918 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.603885 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9b1c610e-c2ad-4c0e-8aeb-10516ced6b9c:/dev/nvme0n1p4 b1b70af9-d935-49de-8e0f-ed5646c7fd5b:/dev/nvme0n1p3] Jul 21 20:14:05.604007 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.603916 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 20:14:05.608883 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.608767 2567 manager.go:216] Machine: {Timestamp:2026-07-21 20:14:05.607598451 +0000 UTC m=+0.287724879 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101518 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec243404529e78222cc5c814e3636614 SystemUUID:ec243404-529e-7822-2cc5-c814e3636614 BootID:e5588f76-ccdb-4edc-9440-d2ef6105e512 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:40:00:f3:ac:b3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:40:00:f3:ac:b3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:be:b3:dd:17:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 20:14:05.608883 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.608876 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 20:14:05.609038 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.609025 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 20:14:05.609884 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.609852 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 20:14:05.610053 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.609887 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-132.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 20:14:05.610598 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.610588 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 20:14:05.610630 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.610599 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 20:14:05.610630 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.610607 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 20:14:05.611175 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.611164 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 20:14:05.612418 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.612406 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 20:14:05.614557 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.614543 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 20:14:05.616510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.616498 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 21 20:14:05.616637 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.616628 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 20:14:05.616666 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.616658 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 21 20:14:05.616698 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.616668 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 20:14:05.616698 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.616683 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 20:14:05.616698 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.616693 2567 kubelet.go:398] "Adding apiserver pod source" Jul 21 20:14:05.616792 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.616702 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 20:14:05.617103 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.617087 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 20:14:05.617158 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.617117 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 20:14:05.620327 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.620305 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 20:14:05.622175 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.622162 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 20:14:05.622218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.622192 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 20:14:05.623199 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623177 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 20:14:05.623199 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623200 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623212 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623222 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623235 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623245 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623257 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623266 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623276 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623286 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623298 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 20:14:05.623344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623306 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 20:14:05.623884 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623872 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 20:14:05.623916 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.623885 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 20:14:05.627245 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.627230 2567 server.go:1268] "Started kubelet" Jul 21 20:14:05.630237 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.630185 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 20:14:05.630463 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.630439 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-132.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 20:14:05.630821 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.630782 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 20:14:05.630890 ip-10-0-141-132 systemd[1]: Started Kubernetes Kubelet. Jul 21 20:14:05.631108 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.631050 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 20:14:05.631108 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.627441 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 20:14:05.631210 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.631163 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 20:14:05.633282 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.633259 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 20:14:05.633874 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.633849 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 20:14:05.634924 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.634909 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 21 20:14:05.636215 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.635452 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-132.ec2.internal.18c46738b6bc9bc0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-132.ec2.internal,UID:ip-10-0-141-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-132.ec2.internal,},FirstTimestamp:2026-07-21 20:14:05.627202496 +0000 UTC m=+0.307328924,LastTimestamp:2026-07-21 20:14:05.627202496 +0000 UTC m=+0.307328924,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-132.ec2.internal,}" Jul 21 20:14:05.639720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.639697 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 20:14:05.639720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.639707 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 20:14:05.639873 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.639786 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:05.640277 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.640245 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 20:14:05.640355 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.640245 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 20:14:05.640355 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.640297 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 20:14:05.640456 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.640424 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 21 20:14:05.640456 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.640432 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 21 20:14:05.640645 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.640626 2567 factory.go:55] Registering systemd factory Jul 21 20:14:05.640709 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.640669 2567 factory.go:223] Registration of the systemd container factory successfully Jul 21 20:14:05.640927 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.640910 2567 factory.go:153] Registering CRI-O factory Jul 21 20:14:05.640927 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.640928 2567 factory.go:223] Registration of the crio container factory successfully Jul 21 20:14:05.641097 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.641011 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 20:14:05.641097 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.641039 2567 factory.go:103] Registering Raw factory Jul 21 20:14:05.641097 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.641054 2567 manager.go:1196] Started watching for new ooms in manager Jul 21 20:14:05.641451 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.641437 2567 manager.go:349] Starting recovery of all containers Jul 21 20:14:05.642497 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.642473 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 20:14:05.642653 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.642636 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-132.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 20:14:05.652870 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.652847 2567 manager.go:354] Recovery completed Jul 21 20:14:05.657501 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.657486 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 20:14:05.659462 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.659442 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientMemory" Jul 21 20:14:05.659544 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.659476 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 20:14:05.659544 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.659488 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientPID" Jul 21 20:14:05.660204 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.660106 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 20:14:05.660204 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.660118 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 20:14:05.660204 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.660122 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 20:14:05.660204 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.660134 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 21 20:14:05.660870 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.660852 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientMemory" Jul 21 20:14:05.660957 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.660885 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 20:14:05.660957 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.660901 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientPID" Jul 21 20:14:05.662670 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.662364 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-132.ec2.internal.18c46738b8a8d9fb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-132.ec2.internal,UID:ip-10-0-141-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-132.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-132.ec2.internal,},FirstTimestamp:2026-07-21 20:14:05.659462139 +0000 UTC m=+0.339588567,LastTimestamp:2026-07-21 20:14:05.659462139 +0000 UTC m=+0.339588567,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-132.ec2.internal,}" Jul 21 20:14:05.664207 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.664093 2567 policy_none.go:49] "None policy: Start" Jul 21 20:14:05.664285 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.664223 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 20:14:05.664285 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.664236 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 20:14:05.666209 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.666196 2567 policy_none.go:47] "Start" Jul 21 20:14:05.672347 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.672177 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-132.ec2.internal.18c46738b8a92660 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-132.ec2.internal,UID:ip-10-0-141-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-132.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-132.ec2.internal,},FirstTimestamp:2026-07-21 20:14:05.659481696 +0000 UTC m=+0.339608124,LastTimestamp:2026-07-21 20:14:05.659481696 +0000 UTC m=+0.339608124,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-132.ec2.internal,}" Jul 21 20:14:05.673271 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.673251 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vh2dh" Jul 21 20:14:05.681760 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.681737 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vh2dh" Jul 21 20:14:05.682587 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.682523 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-132.ec2.internal.18c46738b8a974da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-132.ec2.internal,UID:ip-10-0-141-132.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-132.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-132.ec2.internal,},FirstTimestamp:2026-07-21 20:14:05.659501786 +0000 UTC m=+0.339628214,LastTimestamp:2026-07-21 20:14:05.659501786 +0000 UTC m=+0.339628214,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-132.ec2.internal,}" Jul 21 20:14:05.700566 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.700545 2567 manager.go:337] "Starting Device Plugin manager" Jul 21 20:14:05.700672 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.700587 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 20:14:05.700672 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.700600 2567 server.go:87] "Starting device plugin registration server" Jul 21 20:14:05.700903 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.700892 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 20:14:05.701008 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.700906 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 20:14:05.701057 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.701006 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 20:14:05.701140 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.701123 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 20:14:05.701140 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.701134 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 20:14:05.701665 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.701644 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 20:14:05.701738 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.701688 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:05.797402 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.797369 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 20:14:05.799473 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.798395 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 20:14:05.799473 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.798425 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 20:14:05.799473 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.798445 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 20:14:05.799473 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.798477 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 20:14:05.800863 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.800835 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 20:14:05.800992 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.800978 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 20:14:05.801690 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.801675 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientMemory" Jul 21 20:14:05.801757 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.801707 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 20:14:05.801757 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.801722 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientPID" Jul 21 20:14:05.801757 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.801750 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-132.ec2.internal" Jul 21 20:14:05.811133 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.811069 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-132.ec2.internal" Jul 21 20:14:05.811133 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.811099 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-132.ec2.internal\": node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:05.837639 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.837611 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:05.899090 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.899055 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal"] Jul 21 20:14:05.899216 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.899148 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 20:14:05.900516 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.900501 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientMemory" Jul 21 20:14:05.900603 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.900529 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 20:14:05.900603 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.900540 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientPID" Jul 21 20:14:05.902705 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.902693 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 20:14:05.902814 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.902796 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" Jul 21 20:14:05.902865 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.902831 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 20:14:05.903373 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.903350 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientMemory" Jul 21 20:14:05.903373 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.903359 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientMemory" Jul 21 20:14:05.903494 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.903377 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 20:14:05.903494 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.903390 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientPID" Jul 21 20:14:05.903494 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.903401 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 20:14:05.903494 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.903420 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientPID" Jul 21 20:14:05.905442 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.905426 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal" Jul 21 20:14:05.905515 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.905450 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 20:14:05.906579 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.906563 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientMemory" Jul 21 20:14:05.906662 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.906590 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 20:14:05.906662 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.906599 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientPID" Jul 21 20:14:05.927695 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.927666 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-132.ec2.internal\" not found" node="ip-10-0-141-132.ec2.internal" Jul 21 20:14:05.931910 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.931891 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-132.ec2.internal\" not found" node="ip-10-0-141-132.ec2.internal" Jul 21 20:14:05.937901 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:05.937883 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:05.942163 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.942143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2c9637aa4743415a87fac36d894b9a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal\" (UID: \"e2c9637aa4743415a87fac36d894b9a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" Jul 21 20:14:05.942210 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.942173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2c9637aa4743415a87fac36d894b9a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal\" (UID: \"e2c9637aa4743415a87fac36d894b9a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" Jul 21 20:14:05.942210 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:05.942191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfc1999bccbcd5a41204f2e93a96d415-config\") pod \"kube-apiserver-proxy-ip-10-0-141-132.ec2.internal\" (UID: \"bfc1999bccbcd5a41204f2e93a96d415\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal" Jul 21 20:14:06.038589 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.038561 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:06.042966 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.042929 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfc1999bccbcd5a41204f2e93a96d415-config\") pod \"kube-apiserver-proxy-ip-10-0-141-132.ec2.internal\" (UID: \"bfc1999bccbcd5a41204f2e93a96d415\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal" Jul 21 20:14:06.043019 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.042981 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2c9637aa4743415a87fac36d894b9a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal\" (UID: \"e2c9637aa4743415a87fac36d894b9a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" Jul 21 20:14:06.043019 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.042999 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2c9637aa4743415a87fac36d894b9a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal\" (UID: \"e2c9637aa4743415a87fac36d894b9a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" Jul 21 20:14:06.043076 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.043035 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bfc1999bccbcd5a41204f2e93a96d415-config\") pod \"kube-apiserver-proxy-ip-10-0-141-132.ec2.internal\" (UID: \"bfc1999bccbcd5a41204f2e93a96d415\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal" Jul 21 20:14:06.043076 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.043048 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2c9637aa4743415a87fac36d894b9a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal\" (UID: \"e2c9637aa4743415a87fac36d894b9a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" Jul 21 20:14:06.043133 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.043035 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2c9637aa4743415a87fac36d894b9a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal\" (UID: \"e2c9637aa4743415a87fac36d894b9a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" Jul 21 20:14:06.138958 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.138886 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:06.231347 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.231316 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" Jul 21 20:14:06.234454 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.234430 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal" Jul 21 20:14:06.239303 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.239286 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:06.339651 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.339613 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:06.439983 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.439919 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:06.540374 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.540339 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:06.582616 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.582584 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 20:14:06.583023 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.582733 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 20:14:06.640611 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.640582 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:06.689044 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.688993 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 20:09:05 +0000 UTC" deadline="2028-02-10 22:52:20.996951601 +0000 UTC" Jul 21 20:14:06.689044 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.689039 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13658h38m14.307918155s" Jul 21 20:14:06.735725 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:06.735691 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc1999bccbcd5a41204f2e93a96d415.slice/crio-23b317edd9100906c8b127e3ee0228638b7f911d1d84aa9998d95eac42913d2b WatchSource:0}: Error finding container 23b317edd9100906c8b127e3ee0228638b7f911d1d84aa9998d95eac42913d2b: Status 404 returned error can't find the container with id 23b317edd9100906c8b127e3ee0228638b7f911d1d84aa9998d95eac42913d2b Jul 21 20:14:06.736120 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:06.736101 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c9637aa4743415a87fac36d894b9a7.slice/crio-8c8878e0dd67d567b038e372c8a35a9c7ccdb478d0176fe3758d5b66d297131f WatchSource:0}: Error finding container 8c8878e0dd67d567b038e372c8a35a9c7ccdb478d0176fe3758d5b66d297131f: Status 404 returned error can't find the container with id 8c8878e0dd67d567b038e372c8a35a9c7ccdb478d0176fe3758d5b66d297131f Jul 21 20:14:06.740396 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.740382 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 20:14:06.740643 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.740629 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:06.802031 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.801980 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal" event={"ID":"bfc1999bccbcd5a41204f2e93a96d415","Type":"ContainerStarted","Data":"23b317edd9100906c8b127e3ee0228638b7f911d1d84aa9998d95eac42913d2b"} Jul 21 20:14:06.803543 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.803521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" event={"ID":"e2c9637aa4743415a87fac36d894b9a7","Type":"ContainerStarted","Data":"8c8878e0dd67d567b038e372c8a35a9c7ccdb478d0176fe3758d5b66d297131f"} Jul 21 20:14:06.817465 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.817439 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 20:14:06.841369 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.841340 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:06.865584 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.865565 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 20:14:06.866742 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.866726 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientMemory" Jul 21 20:14:06.866822 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.866753 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 20:14:06.866822 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:06.866762 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeHasSufficientPID" Jul 21 20:14:06.941552 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:06.941520 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:07.024412 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.024333 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 20:14:07.042412 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.042382 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-132.ec2.internal\" not found" Jul 21 20:14:07.100332 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.100300 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 20:14:07.140632 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.140601 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" Jul 21 20:14:07.154906 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.154879 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 20:14:07.155699 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.155684 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal" Jul 21 20:14:07.164437 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.164414 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 20:14:07.617963 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.617914 2567 apiserver.go:52] "Watching apiserver" Jul 21 20:14:07.625056 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.625022 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 20:14:07.626328 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.626295 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rwgp8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w","openshift-cluster-node-tuning-operator/tuned-hfmmh","openshift-dns/node-resolver-jfpqq","openshift-image-registry/node-ca-4dc2n","openshift-multus/multus-additional-cni-plugins-kmc49","openshift-multus/network-metrics-daemon-f57lq","openshift-network-diagnostics/network-check-target-fkdcg","kube-system/global-pull-secret-syncer-zps6q","kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal","openshift-multus/multus-22qgs","openshift-network-operator/iptables-alerter-mt94h","openshift-ovn-kubernetes/ovnkube-node-nzmdl"] Jul 21 20:14:07.629204 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.629179 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.631633 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.631606 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.632506 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.632485 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 20:14:07.632506 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.632499 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 20:14:07.632659 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.632511 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 20:14:07.632659 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.632608 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 20:14:07.632760 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.632673 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-j77wv\"" Jul 21 20:14:07.634505 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.633849 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.634505 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.634238 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 20:14:07.634505 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.634350 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 20:14:07.634505 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.634350 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6lnsg\"" Jul 21 20:14:07.634505 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.634424 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 20:14:07.634786 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.634638 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 20:14:07.634786 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.634736 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 20:14:07.634884 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.634828 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 20:14:07.637396 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.636253 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.637396 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.636615 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 20:14:07.637396 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.636622 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kt2f2\"" Jul 21 20:14:07.637396 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.636964 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 20:14:07.638368 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.638191 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 20:14:07.642213 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.639284 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.642213 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.641178 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 20:14:07.642213 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.641284 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 20:14:07.642213 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.641438 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 20:14:07.642213 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.641612 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k69kk\"" Jul 21 20:14:07.644398 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.644376 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 20:14:07.644661 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.644641 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 20:14:07.645595 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.644952 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 20:14:07.645595 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.645247 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8p957\"" Jul 21 20:14:07.645736 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.645655 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.647901 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.646372 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:07.647901 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.646474 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:07.649703 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.648819 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mgd2v\"" Jul 21 20:14:07.649703 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.649221 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 20:14:07.649703 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.649378 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 20:14:07.649703 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.649575 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:07.649703 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.649640 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:07.650447 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650428 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-hostroot\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.650529 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650462 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-tuned\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.650529 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650490 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-system-cni-dir\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.650529 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650515 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-os-release\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.650680 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650546 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-socket-dir-parent\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.650680 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650569 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-run-openvswitch\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.650680 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650595 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.650680 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650618 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4w8x\" (UniqueName: \"kubernetes.io/projected/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-kube-api-access-f4w8x\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.650680 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650644 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdpkx\" (UniqueName: \"kubernetes.io/projected/a888d619-b063-4711-86a0-1334ca5db69d-kube-api-access-hdpkx\") pod \"iptables-alerter-mt94h\" (UID: \"a888d619-b063-4711-86a0-1334ca5db69d\") " pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.650680 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650667 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650690 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-var-lib-cni-bin\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650712 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-var-lib-cni-multus\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650734 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-log-socket\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650755 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-sys\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a888d619-b063-4711-86a0-1334ca5db69d-iptables-alerter-script\") pod \"iptables-alerter-mt94h\" (UID: \"a888d619-b063-4711-86a0-1334ca5db69d\") " pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650801 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32f0ea72-8c4a-4787-bb7f-b6aa0991eef2-host\") pod \"node-ca-4dc2n\" (UID: \"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2\") " pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650828 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee80a941-0e8e-48c3-8bb9-f29e728f895b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650854 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-etc-kubernetes\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650876 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-sysconfig\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650898 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-systemd\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650920 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-var-lib-kubelet\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650957 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-tmp\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.650977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.650980 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-os-release\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-cni-binary-copy\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-daemon-config\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651055 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-cni-netd\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651078 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-run-k8s-cni-cncf-io\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651101 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-run-multus-certs\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651125 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-sysctl-conf\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651149 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee80a941-0e8e-48c3-8bb9-f29e728f895b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651173 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-run-netns\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651196 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-systemd-units\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-run-systemd\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651241 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-cnibin\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pthj\" (UniqueName: \"kubernetes.io/projected/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-kube-api-access-5pthj\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651321 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-ovnkube-config\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qrw\" (UniqueName: \"kubernetes.io/projected/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-kube-api-access-n9qrw\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651383 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-modprobe-d\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-lib-modules\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.651604 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651428 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-host\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651451 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdpk\" (UniqueName: \"kubernetes.io/projected/32f0ea72-8c4a-4787-bb7f-b6aa0991eef2-kube-api-access-gmdpk\") pod \"node-ca-4dc2n\" (UID: \"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2\") " pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651476 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-etc-openvswitch\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651505 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-sysctl-d\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651528 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-conf-dir\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651551 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-cni-bin\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651574 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-kubernetes\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651601 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a888d619-b063-4711-86a0-1334ca5db69d-host-slash\") pod \"iptables-alerter-mt94h\" (UID: \"a888d619-b063-4711-86a0-1334ca5db69d\") " pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651625 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-run-netns\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651648 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-env-overrides\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-run\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651698 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee80a941-0e8e-48c3-8bb9-f29e728f895b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651732 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srkm6\" (UniqueName: \"kubernetes.io/projected/ee80a941-0e8e-48c3-8bb9-f29e728f895b-kube-api-access-srkm6\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651758 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-var-lib-kubelet\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651782 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-slash\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651820 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.652495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651847 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-ovn-node-metrics-cert\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651871 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-system-cni-dir\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651896 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-cni-dir\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651920 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-var-lib-openvswitch\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651962 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-ovnkube-script-lib\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.651985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-cnibin\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.652007 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-kubelet\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.652035 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-run-ovn\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.652059 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-node-log\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.652097 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/32f0ea72-8c4a-4787-bb7f-b6aa0991eef2-serviceca\") pod \"node-ca-4dc2n\" (UID: \"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2\") " pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.653113 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.652711 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.654781 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.654321 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 20:14:07.654781 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.654564 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:07.654781 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.654640 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:07.656283 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.655851 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c6nth\"" Jul 21 20:14:07.656283 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.656040 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 20:14:07.656283 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.656090 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 20:14:07.658831 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.658104 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:07.661345 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.660640 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 20:14:07.661345 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.660787 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g694g\"" Jul 21 20:14:07.661345 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.661145 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 20:14:07.662239 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.661801 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.666034 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.665434 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 20:14:07.666034 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.665468 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 20:14:07.666034 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.665717 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 20:14:07.666034 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.665860 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4fm8r\"" Jul 21 20:14:07.682126 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.681640 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgf62" Jul 21 20:14:07.691469 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.691097 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zgf62" Jul 21 20:14:07.742130 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.742100 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 20:14:07.752925 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.752813 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-sysctl-d\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.753123 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.752989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1-agent-certs\") pod \"konnectivity-agent-rwgp8\" (UID: \"5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1\") " pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:07.753123 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753040 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1-konnectivity-ca\") pod \"konnectivity-agent-rwgp8\" (UID: \"5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1\") " pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:07.753123 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753068 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec48f297-40e1-4b50-a162-d46157ba43da-kubelet-config\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:07.753123 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-conf-dir\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.753123 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753100 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-sysctl-d\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753135 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-cni-bin\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753171 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-kubernetes\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753200 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-conf-dir\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753203 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a888d619-b063-4711-86a0-1334ca5db69d-host-slash\") pod \"iptables-alerter-mt94h\" (UID: \"a888d619-b063-4711-86a0-1334ca5db69d\") " pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-run-netns\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753247 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a888d619-b063-4711-86a0-1334ca5db69d-host-slash\") pod \"iptables-alerter-mt94h\" (UID: \"a888d619-b063-4711-86a0-1334ca5db69d\") " pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-env-overrides\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753298 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-cni-bin\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-run\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.753371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753362 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee80a941-0e8e-48c3-8bb9-f29e728f895b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753387 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-run\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srkm6\" (UniqueName: \"kubernetes.io/projected/ee80a941-0e8e-48c3-8bb9-f29e728f895b-kube-api-access-srkm6\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753445 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-run-netns\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753475 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-var-lib-kubelet\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753516 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec48f297-40e1-4b50-a162-d46157ba43da-dbus\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753554 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgj5\" (UniqueName: \"kubernetes.io/projected/7ddc2241-beee-4890-bc43-2ca958274867-kube-api-access-4jgj5\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-slash\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753615 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753661 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-ovn-node-metrics-cert\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753689 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e05dd41-bfb8-4bd8-bfe4-955dce5591f4-tmp-dir\") pod \"node-resolver-jfpqq\" (UID: \"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4\") " pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753717 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-system-cni-dir\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753749 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-cni-dir\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753722 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-kubernetes\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.753834 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753820 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-var-lib-openvswitch\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.754496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753892 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-var-lib-openvswitch\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.754496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.753967 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-var-lib-kubelet\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.754496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-slash\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.754496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754105 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-env-overrides\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.754496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754104 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.754496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754195 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-system-cni-dir\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.754496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754342 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-cni-dir\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.754496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754467 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee80a941-0e8e-48c3-8bb9-f29e728f895b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.754856 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754514 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 20:14:07.754856 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-registration-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.754856 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754630 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-ovnkube-script-lib\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.754856 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754659 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-cnibin\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.754856 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754717 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-cnibin\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.754856 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754735 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-kubelet\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755126 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754897 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-run-ovn\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755126 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.754965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-node-log\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755126 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755083 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e05dd41-bfb8-4bd8-bfe4-955dce5591f4-hosts-file\") pod \"node-resolver-jfpqq\" (UID: \"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4\") " pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.755126 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755119 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/32f0ea72-8c4a-4787-bb7f-b6aa0991eef2-serviceca\") pod \"node-ca-4dc2n\" (UID: \"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2\") " pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.755285 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755149 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.755285 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755179 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-hostroot\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.755285 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755215 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-tuned\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.755285 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755242 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:07.755285 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755272 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-system-cni-dir\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.755490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755334 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-os-release\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.755490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755349 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-ovnkube-script-lib\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-socket-dir-parent\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.755490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755388 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-run-openvswitch\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755433 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-kubelet\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755501 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-run-ovn\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755533 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4w8x\" (UniqueName: \"kubernetes.io/projected/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-kube-api-access-f4w8x\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdpkx\" (UniqueName: \"kubernetes.io/projected/a888d619-b063-4711-86a0-1334ca5db69d-kube-api-access-hdpkx\") pod \"iptables-alerter-mt94h\" (UID: \"a888d619-b063-4711-86a0-1334ca5db69d\") " pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755553 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-hostroot\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755615 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-var-lib-cni-bin\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-var-lib-cni-multus\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-log-socket\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.755751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755754 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-sys\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755791 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a888d619-b063-4711-86a0-1334ca5db69d-iptables-alerter-script\") pod \"iptables-alerter-mt94h\" (UID: \"a888d619-b063-4711-86a0-1334ca5db69d\") " pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755819 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32f0ea72-8c4a-4787-bb7f-b6aa0991eef2-host\") pod \"node-ca-4dc2n\" (UID: \"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2\") " pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755886 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrvl4\" (UniqueName: \"kubernetes.io/projected/011ea63a-4a29-4f3e-9710-4b9b299ba69c-kube-api-access-wrvl4\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.755967 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee80a941-0e8e-48c3-8bb9-f29e728f895b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756008 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-etc-kubernetes\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756030 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-sysconfig\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756055 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/32f0ea72-8c4a-4787-bb7f-b6aa0991eef2-serviceca\") pod \"node-ca-4dc2n\" (UID: \"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2\") " pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756055 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-systemd\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756098 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-var-lib-kubelet\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756125 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-tmp\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756151 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:07.756218 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756198 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-socket-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-os-release\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756274 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-cni-binary-copy\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756339 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-daemon-config\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-cni-netd\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756449 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-device-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756501 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-sys-fs\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756546 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhk8\" (UniqueName: \"kubernetes.io/projected/6e05dd41-bfb8-4bd8-bfe4-955dce5591f4-kube-api-access-kfhk8\") pod \"node-resolver-jfpqq\" (UID: \"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4\") " pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-run-k8s-cni-cncf-io\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-run-multus-certs\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756683 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-sysctl-conf\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee80a941-0e8e-48c3-8bb9-f29e728f895b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.756789 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756745 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-run-netns\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756800 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-systemd-units\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-run-systemd\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756872 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-etc-selinux\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756905 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-cnibin\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.756973 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pthj\" (UniqueName: \"kubernetes.io/projected/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-kube-api-access-5pthj\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757014 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-ovnkube-config\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qrw\" (UniqueName: \"kubernetes.io/projected/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-kube-api-access-n9qrw\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757089 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-modprobe-d\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757106 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee80a941-0e8e-48c3-8bb9-f29e728f895b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-lib-modules\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-host\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757206 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-system-cni-dir\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757219 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdpk\" (UniqueName: \"kubernetes.io/projected/32f0ea72-8c4a-4787-bb7f-b6aa0991eef2-kube-api-access-gmdpk\") pod \"node-ca-4dc2n\" (UID: \"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2\") " pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-etc-openvswitch\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.757369 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-os-release\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.758061 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757380 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-sys\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.758061 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757387 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-etc-kubernetes\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.758061 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757514 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-node-log\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.758061 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.757647 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-os-release\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.758061 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758000 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-host-cni-netd\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.758061 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758020 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a888d619-b063-4711-86a0-1334ca5db69d-iptables-alerter-script\") pod \"iptables-alerter-mt94h\" (UID: \"a888d619-b063-4711-86a0-1334ca5db69d\") " pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.758380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758084 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-sysconfig\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.758380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758158 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-systemd\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.758380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758207 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-run-netns\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.758380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758222 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-var-lib-kubelet\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.758380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758284 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-systemd-units\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.758380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758292 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-cni-binary-copy\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.758380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758352 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-cnibin\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.759041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758836 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-daemon-config\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.759041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-run-systemd\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.759041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.758999 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-run-k8s-cni-cncf-io\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.759041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759023 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-lib-modules\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.759356 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-run-multus-certs\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.759356 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759152 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-host\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.759356 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759209 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-sysctl-conf\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.759501 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759360 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-modprobe-d\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.759501 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759451 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-etc-openvswitch\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.759583 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759521 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-multus-socket-dir-parent\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.759666 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759583 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-run-openvswitch\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.759747 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759726 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee80a941-0e8e-48c3-8bb9-f29e728f895b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.760720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee80a941-0e8e-48c3-8bb9-f29e728f895b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.760720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.759922 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-var-lib-cni-multus\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.760720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.760181 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-etc-tuned\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.760720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.760296 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-ovnkube-config\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.760720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.760359 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-ovn-node-metrics-cert\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.761346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.761160 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-log-socket\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.761346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.761243 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-host-var-lib-cni-bin\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.761346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.761298 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32f0ea72-8c4a-4787-bb7f-b6aa0991eef2-host\") pod \"node-ca-4dc2n\" (UID: \"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2\") " pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.762169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.762127 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-tmp\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.768815 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.768722 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srkm6\" (UniqueName: \"kubernetes.io/projected/ee80a941-0e8e-48c3-8bb9-f29e728f895b-kube-api-access-srkm6\") pod \"multus-additional-cni-plugins-kmc49\" (UID: \"ee80a941-0e8e-48c3-8bb9-f29e728f895b\") " pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.768815 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.768728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdpkx\" (UniqueName: \"kubernetes.io/projected/a888d619-b063-4711-86a0-1334ca5db69d-kube-api-access-hdpkx\") pod \"iptables-alerter-mt94h\" (UID: \"a888d619-b063-4711-86a0-1334ca5db69d\") " pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.769424 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.769180 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pthj\" (UniqueName: \"kubernetes.io/projected/4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d-kube-api-access-5pthj\") pod \"multus-22qgs\" (UID: \"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d\") " pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.769424 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.769212 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qrw\" (UniqueName: \"kubernetes.io/projected/fa25280b-6271-4fb4-9961-a9ecdb0ba5ab-kube-api-access-n9qrw\") pod \"ovnkube-node-nzmdl\" (UID: \"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.770872 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.770848 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdpk\" (UniqueName: \"kubernetes.io/projected/32f0ea72-8c4a-4787-bb7f-b6aa0991eef2-kube-api-access-gmdpk\") pod \"node-ca-4dc2n\" (UID: \"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2\") " pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.771802 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.771760 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4w8x\" (UniqueName: \"kubernetes.io/projected/f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5-kube-api-access-f4w8x\") pod \"tuned-hfmmh\" (UID: \"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.858031 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.857993 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:07.858031 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858038 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrvl4\" (UniqueName: \"kubernetes.io/projected/011ea63a-4a29-4f3e-9710-4b9b299ba69c-kube-api-access-wrvl4\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:07.858329 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:07.858329 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858090 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-socket-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858329 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.858163 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:07.858329 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.858233 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:07.858329 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858245 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-socket-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858329 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.858251 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret podName:ec48f297-40e1-4b50-a162-d46157ba43da nodeName:}" failed. No retries permitted until 2026-07-21 20:14:08.358210064 +0000 UTC m=+3.038336499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret") pod "global-pull-secret-syncer-zps6q" (UID: "ec48f297-40e1-4b50-a162-d46157ba43da") : object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:07.858329 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858287 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:07.858329 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.858307 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:08.358290306 +0000 UTC m=+3.038416744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-device-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858375 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-sys-fs\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhk8\" (UniqueName: \"kubernetes.io/projected/6e05dd41-bfb8-4bd8-bfe4-955dce5591f4-kube-api-access-kfhk8\") pod \"node-resolver-jfpqq\" (UID: \"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4\") " pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-etc-selinux\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858439 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-sys-fs\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858431 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-device-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858483 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1-agent-certs\") pod \"konnectivity-agent-rwgp8\" (UID: \"5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1\") " pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858510 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1-konnectivity-ca\") pod \"konnectivity-agent-rwgp8\" (UID: \"5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1\") " pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858537 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec48f297-40e1-4b50-a162-d46157ba43da-kubelet-config\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858550 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-etc-selinux\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec48f297-40e1-4b50-a162-d46157ba43da-dbus\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858599 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgj5\" (UniqueName: \"kubernetes.io/projected/7ddc2241-beee-4890-bc43-2ca958274867-kube-api-access-4jgj5\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858599 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec48f297-40e1-4b50-a162-d46157ba43da-kubelet-config\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858629 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e05dd41-bfb8-4bd8-bfe4-955dce5591f4-tmp-dir\") pod \"node-resolver-jfpqq\" (UID: \"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4\") " pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-registration-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858667 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec48f297-40e1-4b50-a162-d46157ba43da-dbus\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:07.858732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858699 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e05dd41-bfb8-4bd8-bfe4-955dce5591f4-hosts-file\") pod \"node-resolver-jfpqq\" (UID: \"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4\") " pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.859527 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858730 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.859527 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858845 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.859527 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858852 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ddc2241-beee-4890-bc43-2ca958274867-registration-dir\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.859527 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858891 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e05dd41-bfb8-4bd8-bfe4-955dce5591f4-hosts-file\") pod \"node-resolver-jfpqq\" (UID: \"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4\") " pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.859527 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.858998 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e05dd41-bfb8-4bd8-bfe4-955dce5591f4-tmp-dir\") pod \"node-resolver-jfpqq\" (UID: \"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4\") " pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.859527 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.859137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1-konnectivity-ca\") pod \"konnectivity-agent-rwgp8\" (UID: \"5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1\") " pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:07.861379 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.861357 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1-agent-certs\") pod \"konnectivity-agent-rwgp8\" (UID: \"5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1\") " pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:07.872666 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.872575 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 20:14:07.872666 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.872604 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 20:14:07.872666 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.872620 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vmxl for pod openshift-network-diagnostics/network-check-target-fkdcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:07.872899 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:07.872692 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl podName:55c02ab9-c8c4-451c-867b-9090ac46221c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:08.372672961 +0000 UTC m=+3.052799398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7vmxl" (UniqueName: "kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl") pod "network-check-target-fkdcg" (UID: "55c02ab9-c8c4-451c-867b-9090ac46221c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:07.875268 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.875212 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgj5\" (UniqueName: \"kubernetes.io/projected/7ddc2241-beee-4890-bc43-2ca958274867-kube-api-access-4jgj5\") pod \"aws-ebs-csi-driver-node-2jx4w\" (UID: \"7ddc2241-beee-4890-bc43-2ca958274867\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:07.875488 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.875461 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhk8\" (UniqueName: \"kubernetes.io/projected/6e05dd41-bfb8-4bd8-bfe4-955dce5591f4-kube-api-access-kfhk8\") pod \"node-resolver-jfpqq\" (UID: \"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4\") " pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:07.875564 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.875485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrvl4\" (UniqueName: \"kubernetes.io/projected/011ea63a-4a29-4f3e-9710-4b9b299ba69c-kube-api-access-wrvl4\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:07.946819 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.946781 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-22qgs" Jul 21 20:14:07.959093 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.958831 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:07.970968 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.970926 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" Jul 21 20:14:07.976564 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.976532 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mt94h" Jul 21 20:14:07.984978 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.984929 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4dc2n" Jul 21 20:14:07.985864 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:07.985833 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d058d8a_65fa_44b5_9e64_5b9d4ac03d3d.slice/crio-f0c37287262744b98cde570af12471a5d2984ba5f861bc462dcc7151e6854983 WatchSource:0}: Error finding container f0c37287262744b98cde570af12471a5d2984ba5f861bc462dcc7151e6854983: Status 404 returned error can't find the container with id f0c37287262744b98cde570af12471a5d2984ba5f861bc462dcc7151e6854983 Jul 21 20:14:07.986801 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:07.986780 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89016cb_ce8b_4c2c_b17c_54ebbf2b52f5.slice/crio-1225126873bc859d276cb1d172ffd8e44a33cde1326fddbc9e455cf174524376 WatchSource:0}: Error finding container 1225126873bc859d276cb1d172ffd8e44a33cde1326fddbc9e455cf174524376: Status 404 returned error can't find the container with id 1225126873bc859d276cb1d172ffd8e44a33cde1326fddbc9e455cf174524376 Jul 21 20:14:07.987853 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.987814 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 20:14:07.988278 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:07.988228 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa25280b_6271_4fb4_9961_a9ecdb0ba5ab.slice/crio-52d67c07303f1764ab1585c6d32adef87936790c3aab68b16f3e16a4ddc762a8 WatchSource:0}: Error finding container 52d67c07303f1764ab1585c6d32adef87936790c3aab68b16f3e16a4ddc762a8: Status 404 returned error can't find the container with id 52d67c07303f1764ab1585c6d32adef87936790c3aab68b16f3e16a4ddc762a8 Jul 21 20:14:07.991912 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:07.991889 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda888d619_b063_4711_86a0_1334ca5db69d.slice/crio-0dc643bc05a44419edd97d87d2dc1f503f03c85f4689431c716f8294b0cfca00 WatchSource:0}: Error finding container 0dc643bc05a44419edd97d87d2dc1f503f03c85f4689431c716f8294b0cfca00: Status 404 returned error can't find the container with id 0dc643bc05a44419edd97d87d2dc1f503f03c85f4689431c716f8294b0cfca00 Jul 21 20:14:07.992032 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:07.992000 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmc49" Jul 21 20:14:07.998068 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:07.998040 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f0ea72_8c4a_4787_bb7f_b6aa0991eef2.slice/crio-e9f84ab9f459bab81088e736a5d4003cf576852e075ef2174d3a5e8d95e80428 WatchSource:0}: Error finding container e9f84ab9f459bab81088e736a5d4003cf576852e075ef2174d3a5e8d95e80428: Status 404 returned error can't find the container with id e9f84ab9f459bab81088e736a5d4003cf576852e075ef2174d3a5e8d95e80428 Jul 21 20:14:08.000418 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.000400 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jfpqq" Jul 21 20:14:08.008230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.008213 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:08.014639 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.014621 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" Jul 21 20:14:08.360978 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.360879 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:08.360978 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.360922 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:08.361139 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:08.361042 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:08.361139 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:08.361046 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:08.361139 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:08.361105 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:09.361092053 +0000 UTC m=+4.041218468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:08.361139 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:08.361118 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret podName:ec48f297-40e1-4b50-a162-d46157ba43da nodeName:}" failed. No retries permitted until 2026-07-21 20:14:09.36111231 +0000 UTC m=+4.041238725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret") pod "global-pull-secret-syncer-zps6q" (UID: "ec48f297-40e1-4b50-a162-d46157ba43da") : object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:08.461705 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.461670 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:08.461878 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:08.461839 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 20:14:08.461878 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:08.461858 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 20:14:08.461878 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:08.461867 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vmxl for pod openshift-network-diagnostics/network-check-target-fkdcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:08.462030 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:08.461928 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl podName:55c02ab9-c8c4-451c-867b-9090ac46221c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:09.4619027 +0000 UTC m=+4.142029116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vmxl" (UniqueName: "kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl") pod "network-check-target-fkdcg" (UID: "55c02ab9-c8c4-451c-867b-9090ac46221c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:08.654113 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:08.654077 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee80a941_0e8e_48c3_8bb9_f29e728f895b.slice/crio-828b2f6f06b9f9e569396bac3c8814fd7f061c3bdadbbfe846715f2471da344f WatchSource:0}: Error finding container 828b2f6f06b9f9e569396bac3c8814fd7f061c3bdadbbfe846715f2471da344f: Status 404 returned error can't find the container with id 828b2f6f06b9f9e569396bac3c8814fd7f061c3bdadbbfe846715f2471da344f Jul 21 20:14:08.661996 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:08.661969 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2eb5f8_a554_4bd8_8c85_0aa097e3e7e1.slice/crio-ddb4142d80f470780e3eed56d6087b7954f4c9418fd0a4545a20621aa836bfcd WatchSource:0}: Error finding container ddb4142d80f470780e3eed56d6087b7954f4c9418fd0a4545a20621aa836bfcd: Status 404 returned error can't find the container with id ddb4142d80f470780e3eed56d6087b7954f4c9418fd0a4545a20621aa836bfcd Jul 21 20:14:08.662743 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:08.662719 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddc2241_beee_4890_bc43_2ca958274867.slice/crio-a1fe5bd9e16f52abe78fd373e6b0240ad7e828e5de5a97bcde1aae809f423aa4 WatchSource:0}: Error finding container a1fe5bd9e16f52abe78fd373e6b0240ad7e828e5de5a97bcde1aae809f423aa4: Status 404 returned error can't find the container with id a1fe5bd9e16f52abe78fd373e6b0240ad7e828e5de5a97bcde1aae809f423aa4 Jul 21 20:14:08.663375 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:08.663353 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e05dd41_bfb8_4bd8_bfe4_955dce5591f4.slice/crio-a946341499de6781766db3fabad9b69c24ccf10ec6517348bc6f3c87f81da2da WatchSource:0}: Error finding container a946341499de6781766db3fabad9b69c24ccf10ec6517348bc6f3c87f81da2da: Status 404 returned error can't find the container with id a946341499de6781766db3fabad9b69c24ccf10ec6517348bc6f3c87f81da2da Jul 21 20:14:08.695092 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.695066 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 20:09:07 +0000 UTC" deadline="2027-12-19 18:59:33.738420508 +0000 UTC" Jul 21 20:14:08.695092 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.695089 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12382h45m25.043334672s" Jul 21 20:14:08.799836 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.799616 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:08.800046 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:08.799973 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:08.812484 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.812449 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal" event={"ID":"bfc1999bccbcd5a41204f2e93a96d415","Type":"ContainerStarted","Data":"d4c25c4de8cc8a3f304dcb0b43cc040607c2b4fc9628ae53564a728c0e0861e6"} Jul 21 20:14:08.817521 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.817319 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfpqq" event={"ID":"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4","Type":"ContainerStarted","Data":"a946341499de6781766db3fabad9b69c24ccf10ec6517348bc6f3c87f81da2da"} Jul 21 20:14:08.822397 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.822369 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" event={"ID":"7ddc2241-beee-4890-bc43-2ca958274867","Type":"ContainerStarted","Data":"a1fe5bd9e16f52abe78fd373e6b0240ad7e828e5de5a97bcde1aae809f423aa4"} Jul 21 20:14:08.828018 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.827969 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rwgp8" event={"ID":"5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1","Type":"ContainerStarted","Data":"ddb4142d80f470780e3eed56d6087b7954f4c9418fd0a4545a20621aa836bfcd"} Jul 21 20:14:08.829606 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.829543 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4dc2n" event={"ID":"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2","Type":"ContainerStarted","Data":"e9f84ab9f459bab81088e736a5d4003cf576852e075ef2174d3a5e8d95e80428"} Jul 21 20:14:08.833322 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.833295 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mt94h" event={"ID":"a888d619-b063-4711-86a0-1334ca5db69d","Type":"ContainerStarted","Data":"0dc643bc05a44419edd97d87d2dc1f503f03c85f4689431c716f8294b0cfca00"} Jul 21 20:14:08.837353 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.837329 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmc49" event={"ID":"ee80a941-0e8e-48c3-8bb9-f29e728f895b","Type":"ContainerStarted","Data":"828b2f6f06b9f9e569396bac3c8814fd7f061c3bdadbbfe846715f2471da344f"} Jul 21 20:14:08.842219 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.842190 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerStarted","Data":"52d67c07303f1764ab1585c6d32adef87936790c3aab68b16f3e16a4ddc762a8"} Jul 21 20:14:08.844522 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.844499 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22qgs" event={"ID":"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d","Type":"ContainerStarted","Data":"f0c37287262744b98cde570af12471a5d2984ba5f861bc462dcc7151e6854983"} Jul 21 20:14:08.846210 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:08.846189 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" event={"ID":"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5","Type":"ContainerStarted","Data":"1225126873bc859d276cb1d172ffd8e44a33cde1326fddbc9e455cf174524376"} Jul 21 20:14:09.368919 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.368829 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:09.368919 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.368894 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:09.369294 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.369065 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:09.369294 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.369128 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:11.369108564 +0000 UTC m=+6.049234982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:09.369527 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.369507 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:09.369584 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.369566 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret podName:ec48f297-40e1-4b50-a162-d46157ba43da nodeName:}" failed. No retries permitted until 2026-07-21 20:14:11.369550999 +0000 UTC m=+6.049677435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret") pod "global-pull-secret-syncer-zps6q" (UID: "ec48f297-40e1-4b50-a162-d46157ba43da") : object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:09.469514 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.469473 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:09.470193 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.469692 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 20:14:09.470193 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.469719 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 20:14:09.470193 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.469731 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vmxl for pod openshift-network-diagnostics/network-check-target-fkdcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:09.470193 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.469800 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl podName:55c02ab9-c8c4-451c-867b-9090ac46221c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:11.469781612 +0000 UTC m=+6.149908031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vmxl" (UniqueName: "kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl") pod "network-check-target-fkdcg" (UID: "55c02ab9-c8c4-451c-867b-9090ac46221c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:09.695594 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.695484 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 20:09:07 +0000 UTC" deadline="2028-04-24 13:13:45.022950082 +0000 UTC" Jul 21 20:14:09.695594 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.695546 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15424h59m35.327408407s" Jul 21 20:14:09.801774 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.801745 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:09.801974 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.801891 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:09.802298 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.802279 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:09.802392 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:09.802373 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:09.869895 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.869858 2567 generic.go:356] "Generic (PLEG): container finished" podID="e2c9637aa4743415a87fac36d894b9a7" containerID="dd28b8afa117a33d90eeaf93a561f9a2b9e0592785ad6674a84f683a158768d3" exitCode=0 Jul 21 20:14:09.870087 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.870017 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" event={"ID":"e2c9637aa4743415a87fac36d894b9a7","Type":"ContainerDied","Data":"dd28b8afa117a33d90eeaf93a561f9a2b9e0592785ad6674a84f683a158768d3"} Jul 21 20:14:09.889219 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:09.889144 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-132.ec2.internal" podStartSLOduration=2.889122232 podStartE2EDuration="2.889122232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:08.829057747 +0000 UTC m=+3.509184186" watchObservedRunningTime="2026-07-21 20:14:09.889122232 +0000 UTC m=+4.569248670" Jul 21 20:14:10.800090 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:10.800061 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:10.800558 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:10.800196 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:10.886742 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:10.886077 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" event={"ID":"e2c9637aa4743415a87fac36d894b9a7","Type":"ContainerStarted","Data":"bc0d420e2e73cb3b497ffcd68072cced146793682380afba3aac3031fa71f8a7"} Jul 21 20:14:10.905403 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:10.905348 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-132.ec2.internal" podStartSLOduration=3.905328727 podStartE2EDuration="3.905328727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:10.904731409 +0000 UTC m=+5.584857848" watchObservedRunningTime="2026-07-21 20:14:10.905328727 +0000 UTC m=+5.585455166" Jul 21 20:14:11.389253 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:11.388443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:11.389253 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:11.388508 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:11.389253 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.388675 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:11.389253 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.388742 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:15.38872215 +0000 UTC m=+10.068848568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:11.389253 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.389156 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:11.389253 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.389212 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret podName:ec48f297-40e1-4b50-a162-d46157ba43da nodeName:}" failed. No retries permitted until 2026-07-21 20:14:15.389195639 +0000 UTC m=+10.069322073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret") pod "global-pull-secret-syncer-zps6q" (UID: "ec48f297-40e1-4b50-a162-d46157ba43da") : object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:11.490442 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:11.489782 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:11.490442 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.490010 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 20:14:11.490442 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.490033 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 20:14:11.490442 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.490046 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vmxl for pod openshift-network-diagnostics/network-check-target-fkdcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:11.490442 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.490107 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl podName:55c02ab9-c8c4-451c-867b-9090ac46221c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:15.490090357 +0000 UTC m=+10.170216780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vmxl" (UniqueName: "kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl") pod "network-check-target-fkdcg" (UID: "55c02ab9-c8c4-451c-867b-9090ac46221c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:11.802270 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:11.801539 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:11.802270 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.801698 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:11.805668 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:11.802170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:11.808431 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:11.806003 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:12.799670 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:12.799624 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:12.799843 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:12.799752 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:13.799694 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:13.798974 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:13.799694 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:13.799003 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:13.799694 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:13.799117 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:13.799694 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:13.799214 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:14.798617 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:14.798583 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:14.798775 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:14.798717 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:15.423256 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:15.423007 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:15.423256 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:15.423099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:15.423256 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.423162 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:15.423256 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.423173 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:15.423256 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.423222 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret podName:ec48f297-40e1-4b50-a162-d46157ba43da nodeName:}" failed. No retries permitted until 2026-07-21 20:14:23.423205542 +0000 UTC m=+18.103331957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret") pod "global-pull-secret-syncer-zps6q" (UID: "ec48f297-40e1-4b50-a162-d46157ba43da") : object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:15.423256 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.423238 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:23.423230103 +0000 UTC m=+18.103356517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:15.523979 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:15.523803 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:15.524162 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.524009 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 20:14:15.524162 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.524040 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 20:14:15.524162 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.524054 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vmxl for pod openshift-network-diagnostics/network-check-target-fkdcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:15.524162 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.524119 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl podName:55c02ab9-c8c4-451c-867b-9090ac46221c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:23.524100388 +0000 UTC m=+18.204226806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vmxl" (UniqueName: "kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl") pod "network-check-target-fkdcg" (UID: "55c02ab9-c8c4-451c-867b-9090ac46221c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:15.801588 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:15.800993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:15.801588 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.801134 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:15.801588 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:15.801235 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:15.801588 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:15.801360 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:16.799061 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:16.799019 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:16.799616 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:16.799160 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:17.799048 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:17.798953 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:17.799272 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:17.799098 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:17.799660 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:17.799432 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:17.799660 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:17.799529 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:18.799240 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:18.799210 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:18.799372 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:18.799341 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:19.799296 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:19.799256 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:19.799464 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:19.799313 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:19.799464 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:19.799412 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:19.799854 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:19.799544 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:20.799605 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:20.799569 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:20.800046 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:20.799698 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:21.799225 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:21.799183 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:21.799461 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:21.799318 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:21.799461 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:21.799353 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:21.799461 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:21.799431 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:22.799069 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:22.799036 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:22.799529 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:22.799148 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:23.484342 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:23.484304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:23.484587 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:23.484352 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:23.484587 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.484465 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:23.484587 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.484470 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:23.484587 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.484527 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:39.484508095 +0000 UTC m=+34.164634515 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:23.484587 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.484547 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret podName:ec48f297-40e1-4b50-a162-d46157ba43da nodeName:}" failed. No retries permitted until 2026-07-21 20:14:39.484537041 +0000 UTC m=+34.164663455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret") pod "global-pull-secret-syncer-zps6q" (UID: "ec48f297-40e1-4b50-a162-d46157ba43da") : object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:23.585595 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:23.585566 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:23.585783 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.585724 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 20:14:23.585783 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.585745 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 20:14:23.585783 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.585759 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vmxl for pod openshift-network-diagnostics/network-check-target-fkdcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:23.585925 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.585832 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl podName:55c02ab9-c8c4-451c-867b-9090ac46221c nodeName:}" failed. No retries permitted until 2026-07-21 20:14:39.585805553 +0000 UTC m=+34.265931991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vmxl" (UniqueName: "kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl") pod "network-check-target-fkdcg" (UID: "55c02ab9-c8c4-451c-867b-9090ac46221c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:23.802486 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:23.802412 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:23.802486 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:23.802440 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:23.802932 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.802524 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:23.802932 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:23.802681 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:24.799527 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:24.799491 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:24.799722 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:24.799646 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:25.803052 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:25.803015 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:25.803467 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:25.803015 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:25.803467 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:25.803145 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:25.803467 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:25.803208 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:26.799173 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:26.799140 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:26.799373 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:26.799258 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:27.799393 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.799170 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:27.799922 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.799227 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:27.799922 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:27.799439 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:27.799922 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:27.799511 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:27.921137 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.920891 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" event={"ID":"f89016cb-ce8b-4c2c-b17c-54ebbf2b52f5","Type":"ContainerStarted","Data":"eaf770507ca3009ca25bd5a098b994e0c0846d6c93a02ffe6d948fac4f9aae06"} Jul 21 20:14:27.922732 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.922704 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jfpqq" event={"ID":"6e05dd41-bfb8-4bd8-bfe4-955dce5591f4","Type":"ContainerStarted","Data":"d11a32842a33dbf4987bae8a5d2a6597aaad7f5dbe7461d074dd8a67ed162bb9"} Jul 21 20:14:27.924066 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.924046 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" event={"ID":"7ddc2241-beee-4890-bc43-2ca958274867","Type":"ContainerStarted","Data":"5ef8938c3d79e2af5672e53039ee9edb2251c0c3c5042b1bb02d5d3dbd24ea0d"} Jul 21 20:14:27.925429 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.925398 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rwgp8" event={"ID":"5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1","Type":"ContainerStarted","Data":"af977f3b6114950c272a1c99d757270dde53bfe06950320be9c35eb73b038776"} Jul 21 20:14:27.926858 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.926840 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4dc2n" event={"ID":"32f0ea72-8c4a-4787-bb7f-b6aa0991eef2","Type":"ContainerStarted","Data":"ea79e1548060b1534c8bbef40d4d8de840c7dd2716a0a5e45c97f228bc70acac"} Jul 21 20:14:27.928260 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.928242 2567 generic.go:356] "Generic (PLEG): container finished" podID="ee80a941-0e8e-48c3-8bb9-f29e728f895b" containerID="b19e3f50d1258c449b861998bd7a5b8a278e4f54003dba38c70f62ba016ba16d" exitCode=0 Jul 21 20:14:27.928341 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.928319 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmc49" event={"ID":"ee80a941-0e8e-48c3-8bb9-f29e728f895b","Type":"ContainerDied","Data":"b19e3f50d1258c449b861998bd7a5b8a278e4f54003dba38c70f62ba016ba16d"} Jul 21 20:14:27.930163 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.930147 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:14:27.930524 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.930500 2567 generic.go:356] "Generic (PLEG): container finished" podID="fa25280b-6271-4fb4-9961-a9ecdb0ba5ab" containerID="11cdf3c081d8e9b5eb4f3b9f42c14623c70fa163fbd74d252eb000cb4929261c" exitCode=1 Jul 21 20:14:27.930591 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.930555 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerStarted","Data":"4abf8a4dab06a9ad180734a0b3b6d6426096596f484ea448e3a505f8c47fa52f"} Jul 21 20:14:27.930635 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.930625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerDied","Data":"11cdf3c081d8e9b5eb4f3b9f42c14623c70fa163fbd74d252eb000cb4929261c"} Jul 21 20:14:27.930690 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.930677 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerStarted","Data":"aa1aa5bf875e41963a4f61dcc68bd5be8700a585c302f82619cd2aff491ac245"} Jul 21 20:14:27.931837 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.931818 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-22qgs" event={"ID":"4d058d8a-65fa-44b5-9e64-5b9d4ac03d3d","Type":"ContainerStarted","Data":"23306205534d5eb548183bcf9d234cafada608a0c0fa4c524929651c1b67c452"} Jul 21 20:14:27.969635 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:27.969594 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hfmmh" podStartSLOduration=3.667310842 podStartE2EDuration="22.969577203s" totalImagesPullingTime="19.302266361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:27.944534206 +0000 UTC m=+22.624660640" watchObservedRunningTime="2026-07-21 20:14:27.969577203 +0000 UTC m=+22.649703654" Jul 21 20:14:28.002350 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.002311 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4dc2n" podStartSLOduration=3.727403706 podStartE2EDuration="23.002295982s" totalImagesPullingTime="19.274892276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:27.969910782 +0000 UTC m=+22.650037218" watchObservedRunningTime="2026-07-21 20:14:28.002295982 +0000 UTC m=+22.682422422" Jul 21 20:14:28.034634 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.034586 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-22qgs" podStartSLOduration=3.697905889 podStartE2EDuration="23.034571033s" totalImagesPullingTime="19.336665144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:28.003128988 +0000 UTC m=+22.683255426" watchObservedRunningTime="2026-07-21 20:14:28.034571033 +0000 UTC m=+22.714697470" Jul 21 20:14:28.087481 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.087438 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jfpqq" podStartSLOduration=3.459845257 podStartE2EDuration="22.08742123s" totalImagesPullingTime="18.627575973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:28.058838839 +0000 UTC m=+22.738965276" watchObservedRunningTime="2026-07-21 20:14:28.08742123 +0000 UTC m=+22.767547728" Jul 21 20:14:28.798686 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.798650 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:28.798877 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:28.798758 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:28.937429 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.937407 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:14:28.937983 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.937855 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerStarted","Data":"d8212a2658e78eb60047ea128a771a6078f98abb443305bd0385c779671dc512"} Jul 21 20:14:28.937983 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.937963 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerStarted","Data":"db6561d5ff93f1ec5b1070a5029daf11801c915f4d7ff25054fd4458a0b21f12"} Jul 21 20:14:28.938098 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.938016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerStarted","Data":"97f6757b6149605b4302c63de311f7ad1b0db52a05b65b3b0147ab5f5c72ed1f"} Jul 21 20:14:28.939641 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.939619 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mt94h" event={"ID":"a888d619-b063-4711-86a0-1334ca5db69d","Type":"ContainerStarted","Data":"7f28bd6022e57b2de5c5bfb4d72238ef9751b742b6366571d1fe6e7e4f76743a"} Jul 21 20:14:28.970028 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.970003 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 20:14:28.972197 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:28.972155 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rwgp8" podStartSLOduration=4.343182621 podStartE2EDuration="22.972138733s" totalImagesPullingTime="18.628956112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:28.087361017 +0000 UTC m=+22.767487469" watchObservedRunningTime="2026-07-21 20:14:28.972138733 +0000 UTC m=+23.652265170" Jul 21 20:14:29.713089 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:29.712975 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T20:14:28.970022497Z","UUID":"9067cad2-d375-494e-8764-382b8ae4dcf0","Handler":null,"Name":"","Endpoint":""} Jul 21 20:14:29.714874 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:29.714848 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 20:14:29.715041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:29.714895 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 20:14:29.799244 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:29.799209 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:29.799407 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:29.799209 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:29.799407 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:29.799342 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:29.799524 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:29.799433 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:29.943788 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:29.943747 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" event={"ID":"7ddc2241-beee-4890-bc43-2ca958274867","Type":"ContainerStarted","Data":"5f2ba55991eb84445e9285482a6f41962b9fd0aa13dacc5668ccbab099d41a7c"} Jul 21 20:14:30.426020 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:30.425989 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:30.427348 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:30.426759 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:30.446332 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:30.446282 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mt94h" podStartSLOduration=6.148460845 podStartE2EDuration="25.446258866s" totalImagesPullingTime="19.297798021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:28.972432931 +0000 UTC m=+23.652559368" watchObservedRunningTime="2026-07-21 20:14:30.446258866 +0000 UTC m=+25.126385305" Jul 21 20:14:30.799272 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:30.799184 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:30.799436 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:30.799319 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:30.949311 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:30.949275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" event={"ID":"7ddc2241-beee-4890-bc43-2ca958274867","Type":"ContainerStarted","Data":"0c201fbdef5bfe19db3f8c5051844863f63dd9dda10cd9a1276b71f459ee9cc6"} Jul 21 20:14:30.952734 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:30.952714 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:14:30.953183 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:30.953159 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerStarted","Data":"ff3c3a175f7ab0e220b625f9b72da84f7100ca6a2c0dda82b763b2cfbc283598"} Jul 21 20:14:31.127838 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:31.127803 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:31.128560 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:31.128539 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rwgp8" Jul 21 20:14:31.146082 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:31.146034 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2jx4w" podStartSLOduration=3.8594867539999997 podStartE2EDuration="25.14601731s" totalImagesPullingTime="21.286530556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:30.979730647 +0000 UTC m=+25.659857085" watchObservedRunningTime="2026-07-21 20:14:31.14601731 +0000 UTC m=+25.826143746" Jul 21 20:14:31.799572 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:31.799534 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:31.799782 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:31.799589 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:31.799782 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:31.799695 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:31.799867 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:31.799820 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:32.799773 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:32.799692 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:32.800294 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:32.799832 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:33.799535 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.799342 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:33.799721 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.799341 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:33.799721 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:33.799623 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:33.799721 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:33.799676 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:33.961080 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.960997 2567 generic.go:356] "Generic (PLEG): container finished" podID="ee80a941-0e8e-48c3-8bb9-f29e728f895b" containerID="b9bfd8596c3454255d80f5629fabe6ee92105a3fa68146fc4ce01cbcc344f620" exitCode=0 Jul 21 20:14:33.961239 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.961077 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmc49" event={"ID":"ee80a941-0e8e-48c3-8bb9-f29e728f895b","Type":"ContainerDied","Data":"b9bfd8596c3454255d80f5629fabe6ee92105a3fa68146fc4ce01cbcc344f620"} Jul 21 20:14:33.966654 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.966633 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:14:33.967088 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.967056 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerStarted","Data":"b0cd9b66a17fb41a0f5d2873a4be9f4082615ade53103c8b3fbe1e5e4b98327a"} Jul 21 20:14:33.967316 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.967296 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:33.967397 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.967379 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:33.967481 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.967430 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:33.967567 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.967549 2567 scope.go:117] "RemoveContainer" containerID="11cdf3c081d8e9b5eb4f3b9f42c14623c70fa163fbd74d252eb000cb4929261c" Jul 21 20:14:33.984008 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.983984 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:33.984198 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:33.984184 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:14:34.798713 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:34.798674 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:34.799017 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:34.798803 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:34.973167 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:34.973143 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:14:34.973552 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:34.973447 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" event={"ID":"fa25280b-6271-4fb4-9961-a9ecdb0ba5ab","Type":"ContainerStarted","Data":"0ef714ff723dcf6d2204fad4ded35a1c1b80f307ce3e73e9c06fe96fa35da287"} Jul 21 20:14:35.018683 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:35.018634 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" podStartSLOduration=10.465863941 podStartE2EDuration="30.018618551s" totalImagesPullingTime="19.55275461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:35.014259266 +0000 UTC m=+29.694385721" watchObservedRunningTime="2026-07-21 20:14:35.018618551 +0000 UTC m=+29.698744987" Jul 21 20:14:35.674039 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:35.673998 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fkdcg"] Jul 21 20:14:35.674202 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:35.674148 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:35.674248 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:35.674233 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:35.674475 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:35.674454 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f57lq"] Jul 21 20:14:35.674559 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:35.674547 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:35.674641 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:35.674625 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:35.678164 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:35.678138 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zps6q"] Jul 21 20:14:35.678296 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:35.678211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:35.678296 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:35.678276 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:35.977245 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:35.977014 2567 generic.go:356] "Generic (PLEG): container finished" podID="ee80a941-0e8e-48c3-8bb9-f29e728f895b" containerID="5af72f84e29cd1beaee2ac62d4a8f6dc81e0c187e274dcd52ada1de8e5bd1b29" exitCode=0 Jul 21 20:14:35.977581 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:35.977102 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmc49" event={"ID":"ee80a941-0e8e-48c3-8bb9-f29e728f895b","Type":"ContainerDied","Data":"5af72f84e29cd1beaee2ac62d4a8f6dc81e0c187e274dcd52ada1de8e5bd1b29"} Jul 21 20:14:37.799351 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:37.799316 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:37.799351 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:37.799359 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:37.800139 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:37.799316 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:37.800139 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:37.799441 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:37.800139 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:37.799495 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:37.800139 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:37.799564 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:37.983632 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:37.983598 2567 generic.go:356] "Generic (PLEG): container finished" podID="ee80a941-0e8e-48c3-8bb9-f29e728f895b" containerID="d5fff41e96483d4f63a38c3dabda2e5e5545d2a9922744ea1a3b69d6defa5b8c" exitCode=0 Jul 21 20:14:37.983818 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:37.983646 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmc49" event={"ID":"ee80a941-0e8e-48c3-8bb9-f29e728f895b","Type":"ContainerDied","Data":"d5fff41e96483d4f63a38c3dabda2e5e5545d2a9922744ea1a3b69d6defa5b8c"} Jul 21 20:14:39.505367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:39.505324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:39.505829 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:39.505384 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:39.505829 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.505497 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:39.505829 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.505504 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:39.505829 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.505561 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:15:11.505541069 +0000 UTC m=+66.185667484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 20:14:39.505829 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.505580 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret podName:ec48f297-40e1-4b50-a162-d46157ba43da nodeName:}" failed. No retries permitted until 2026-07-21 20:15:11.505571329 +0000 UTC m=+66.185697744 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret") pod "global-pull-secret-syncer-zps6q" (UID: "ec48f297-40e1-4b50-a162-d46157ba43da") : object "kube-system"/"original-pull-secret" not registered Jul 21 20:14:39.606045 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:39.606006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:39.606229 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.606212 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 20:14:39.606294 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.606238 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 20:14:39.606294 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.606253 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vmxl for pod openshift-network-diagnostics/network-check-target-fkdcg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:39.606404 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.606323 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl podName:55c02ab9-c8c4-451c-867b-9090ac46221c nodeName:}" failed. No retries permitted until 2026-07-21 20:15:11.606304441 +0000 UTC m=+66.286430877 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vmxl" (UniqueName: "kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl") pod "network-check-target-fkdcg" (UID: "55c02ab9-c8c4-451c-867b-9090ac46221c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 20:14:39.799541 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:39.799486 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:39.799736 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:39.799487 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:39.799736 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.799632 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zps6q" podUID="ec48f297-40e1-4b50-a162-d46157ba43da" Jul 21 20:14:39.799736 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.799717 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fkdcg" podUID="55c02ab9-c8c4-451c-867b-9090ac46221c" Jul 21 20:14:39.799910 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:39.799487 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:39.799910 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:39.799827 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:14:40.121126 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.121100 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-132.ec2.internal" event="NodeReady" Jul 21 20:14:40.121305 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.121255 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 20:14:40.174093 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.174061 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-664d77bf4-dsmcf"] Jul 21 20:14:40.178045 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.178021 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.180769 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.180747 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 20:14:40.181522 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.181500 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 20:14:40.182238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.182221 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 20:14:40.183130 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.183114 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k44nk\"" Jul 21 20:14:40.186786 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.186766 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 20:14:40.195169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.195142 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-664d77bf4-dsmcf"] Jul 21 20:14:40.200593 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.200152 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g29mm"] Jul 21 20:14:40.204221 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.204197 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-czv2k"] Jul 21 20:14:40.204353 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.204336 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.207910 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.207888 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czv2k" Jul 21 20:14:40.208209 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.208171 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 20:14:40.209027 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.209005 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 20:14:40.209300 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.209281 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v6nlp\"" Jul 21 20:14:40.212603 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.211289 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 20:14:40.212603 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.211441 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t5nj5\"" Jul 21 20:14:40.212603 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.211866 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 20:14:40.212603 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.212316 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 20:14:40.225472 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.225448 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g29mm"] Jul 21 20:14:40.243007 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.242974 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czv2k"] Jul 21 20:14:40.312486 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312403 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.312486 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312450 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-image-registry-private-configuration\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.312486 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312477 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtnc\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-kube-api-access-mrtnc\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.312738 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3190751-b8b6-4693-a0bd-ff461895f8a6-cert\") pod \"ingress-canary-czv2k\" (UID: \"a3190751-b8b6-4693-a0bd-ff461895f8a6\") " pod="openshift-ingress-canary/ingress-canary-czv2k" Jul 21 20:14:40.312738 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bde1aba-deb0-4a77-898b-d3a48bcfca67-config-volume\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.312738 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312613 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-installation-pull-secrets\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.312738 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312638 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-bound-sa-token\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.312738 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312679 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7fda0385-d801-453a-ab06-bcac6b323a07-ca-trust-extracted\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.312738 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312704 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.312738 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312723 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-trusted-ca\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.313024 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312751 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bde1aba-deb0-4a77-898b-d3a48bcfca67-tmp-dir\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.313024 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312790 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-registry-certificates\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.313024 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312813 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvpl\" (UniqueName: \"kubernetes.io/projected/9bde1aba-deb0-4a77-898b-d3a48bcfca67-kube-api-access-bpvpl\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.313024 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.312884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srp4\" (UniqueName: \"kubernetes.io/projected/a3190751-b8b6-4693-a0bd-ff461895f8a6-kube-api-access-2srp4\") pod \"ingress-canary-czv2k\" (UID: \"a3190751-b8b6-4693-a0bd-ff461895f8a6\") " pod="openshift-ingress-canary/ingress-canary-czv2k" Jul 21 20:14:40.413744 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.413706 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7fda0385-d801-453a-ab06-bcac6b323a07-ca-trust-extracted\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.413978 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.413750 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.413978 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.413775 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-trusted-ca\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.413978 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.413807 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bde1aba-deb0-4a77-898b-d3a48bcfca67-tmp-dir\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.413978 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.413842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-registry-certificates\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.413978 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.413864 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvpl\" (UniqueName: \"kubernetes.io/projected/9bde1aba-deb0-4a77-898b-d3a48bcfca67-kube-api-access-bpvpl\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.413978 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.413875 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 20:14:40.413978 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.413922 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srp4\" (UniqueName: \"kubernetes.io/projected/a3190751-b8b6-4693-a0bd-ff461895f8a6-kube-api-access-2srp4\") pod \"ingress-canary-czv2k\" (UID: \"a3190751-b8b6-4693-a0bd-ff461895f8a6\") " pod="openshift-ingress-canary/ingress-canary-czv2k" Jul 21 20:14:40.413978 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.413964 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls podName:9bde1aba-deb0-4a77-898b-d3a48bcfca67 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:40.913928656 +0000 UTC m=+35.594055074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls") pod "dns-default-g29mm" (UID: "9bde1aba-deb0-4a77-898b-d3a48bcfca67") : secret "dns-default-metrics-tls" not found Jul 21 20:14:40.414357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414000 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.414357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414033 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-image-registry-private-configuration\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.414357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414060 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtnc\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-kube-api-access-mrtnc\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.414357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414100 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3190751-b8b6-4693-a0bd-ff461895f8a6-cert\") pod \"ingress-canary-czv2k\" (UID: \"a3190751-b8b6-4693-a0bd-ff461895f8a6\") " pod="openshift-ingress-canary/ingress-canary-czv2k" Jul 21 20:14:40.414357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414122 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bde1aba-deb0-4a77-898b-d3a48bcfca67-config-volume\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.414357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414150 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-installation-pull-secrets\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.414357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414174 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-bound-sa-token\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.414357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414211 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7fda0385-d801-453a-ab06-bcac6b323a07-ca-trust-extracted\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.414357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414251 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bde1aba-deb0-4a77-898b-d3a48bcfca67-tmp-dir\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.414779 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-registry-certificates\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.414779 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9bde1aba-deb0-4a77-898b-d3a48bcfca67-config-volume\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.414883 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.414786 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 20:14:40.414883 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.414800 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-664d77bf4-dsmcf: secret "image-registry-tls" not found Jul 21 20:14:40.414883 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.414866 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls podName:7fda0385-d801-453a-ab06-bcac6b323a07 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:40.914849232 +0000 UTC m=+35.594975663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls") pod "image-registry-664d77bf4-dsmcf" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07") : secret "image-registry-tls" not found Jul 21 20:14:40.415055 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.414872 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-trusted-ca\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.418627 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.418602 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-image-registry-private-configuration\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.418757 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.418631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-installation-pull-secrets\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.418757 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.418659 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3190751-b8b6-4693-a0bd-ff461895f8a6-cert\") pod \"ingress-canary-czv2k\" (UID: \"a3190751-b8b6-4693-a0bd-ff461895f8a6\") " pod="openshift-ingress-canary/ingress-canary-czv2k" Jul 21 20:14:40.432294 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.432266 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtnc\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-kube-api-access-mrtnc\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.435533 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.435490 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srp4\" (UniqueName: \"kubernetes.io/projected/a3190751-b8b6-4693-a0bd-ff461895f8a6-kube-api-access-2srp4\") pod \"ingress-canary-czv2k\" (UID: \"a3190751-b8b6-4693-a0bd-ff461895f8a6\") " pod="openshift-ingress-canary/ingress-canary-czv2k" Jul 21 20:14:40.436523 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.436502 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvpl\" (UniqueName: \"kubernetes.io/projected/9bde1aba-deb0-4a77-898b-d3a48bcfca67-kube-api-access-bpvpl\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.436861 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.436841 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-bound-sa-token\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.523266 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.523230 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czv2k" Jul 21 20:14:40.705675 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.705432 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czv2k"] Jul 21 20:14:40.709989 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:14:40.709966 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3190751_b8b6_4693_a0bd_ff461895f8a6.slice/crio-6c4041fcad5d83aff25279ea85ae334f51310d0774b64f10e0921401ad49e0a0 WatchSource:0}: Error finding container 6c4041fcad5d83aff25279ea85ae334f51310d0774b64f10e0921401ad49e0a0: Status 404 returned error can't find the container with id 6c4041fcad5d83aff25279ea85ae334f51310d0774b64f10e0921401ad49e0a0 Jul 21 20:14:40.919248 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.919212 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:40.919416 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.919273 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:40.919416 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.919393 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 20:14:40.919416 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.919396 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 20:14:40.919416 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.919417 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-664d77bf4-dsmcf: secret "image-registry-tls" not found Jul 21 20:14:40.919592 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.919457 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls podName:9bde1aba-deb0-4a77-898b-d3a48bcfca67 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:41.919439751 +0000 UTC m=+36.599566171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls") pod "dns-default-g29mm" (UID: "9bde1aba-deb0-4a77-898b-d3a48bcfca67") : secret "dns-default-metrics-tls" not found Jul 21 20:14:40.919592 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:40.919471 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls podName:7fda0385-d801-453a-ab06-bcac6b323a07 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:41.919465793 +0000 UTC m=+36.599592207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls") pod "image-registry-664d77bf4-dsmcf" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07") : secret "image-registry-tls" not found Jul 21 20:14:40.993026 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:40.992993 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czv2k" event={"ID":"a3190751-b8b6-4693-a0bd-ff461895f8a6","Type":"ContainerStarted","Data":"6c4041fcad5d83aff25279ea85ae334f51310d0774b64f10e0921401ad49e0a0"} Jul 21 20:14:41.799678 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.799638 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:14:41.800213 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.799638 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:14:41.800213 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.799637 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:14:41.803478 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.803455 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 20:14:41.804327 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.804301 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 20:14:41.804431 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.804335 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 20:14:41.804431 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.804374 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bxdt9\"" Jul 21 20:14:41.804431 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.804407 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 20:14:41.804593 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.804485 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7b7qw\"" Jul 21 20:14:41.929097 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.929062 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:41.929265 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:41.929128 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:41.929265 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:41.929200 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 20:14:41.929265 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:41.929217 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-664d77bf4-dsmcf: secret "image-registry-tls" not found Jul 21 20:14:41.929265 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:41.929237 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 20:14:41.929448 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:41.929277 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls podName:7fda0385-d801-453a-ab06-bcac6b323a07 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:43.929258401 +0000 UTC m=+38.609384826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls") pod "image-registry-664d77bf4-dsmcf" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07") : secret "image-registry-tls" not found Jul 21 20:14:41.929448 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:41.929295 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls podName:9bde1aba-deb0-4a77-898b-d3a48bcfca67 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:43.929285962 +0000 UTC m=+38.609412377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls") pod "dns-default-g29mm" (UID: "9bde1aba-deb0-4a77-898b-d3a48bcfca67") : secret "dns-default-metrics-tls" not found Jul 21 20:14:43.001069 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:43.001030 2567 generic.go:356] "Generic (PLEG): container finished" podID="ee80a941-0e8e-48c3-8bb9-f29e728f895b" containerID="b4bf465983fe3e70ed6e49baffa8953f98668c70f14d6a981fb8d235af9a8eb4" exitCode=0 Jul 21 20:14:43.001484 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:43.001081 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmc49" event={"ID":"ee80a941-0e8e-48c3-8bb9-f29e728f895b","Type":"ContainerDied","Data":"b4bf465983fe3e70ed6e49baffa8953f98668c70f14d6a981fb8d235af9a8eb4"} Jul 21 20:14:43.946025 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:43.945982 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:43.946204 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:43.946100 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:43.946204 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:43.946176 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 20:14:43.946204 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:43.946199 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 20:14:43.946335 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:43.946209 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-664d77bf4-dsmcf: secret "image-registry-tls" not found Jul 21 20:14:43.946335 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:43.946250 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls podName:9bde1aba-deb0-4a77-898b-d3a48bcfca67 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:47.946227349 +0000 UTC m=+42.626353778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls") pod "dns-default-g29mm" (UID: "9bde1aba-deb0-4a77-898b-d3a48bcfca67") : secret "dns-default-metrics-tls" not found Jul 21 20:14:43.946335 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:43.946264 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls podName:7fda0385-d801-453a-ab06-bcac6b323a07 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:47.946258289 +0000 UTC m=+42.626384707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls") pod "image-registry-664d77bf4-dsmcf" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07") : secret "image-registry-tls" not found Jul 21 20:14:44.007000 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:44.006799 2567 generic.go:356] "Generic (PLEG): container finished" podID="ee80a941-0e8e-48c3-8bb9-f29e728f895b" containerID="c0e85c8c571657f87657c7cd10f9c50a8af469b7fadab1a3bacb2b1a84abf6bf" exitCode=0 Jul 21 20:14:44.007355 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:44.006876 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmc49" event={"ID":"ee80a941-0e8e-48c3-8bb9-f29e728f895b","Type":"ContainerDied","Data":"c0e85c8c571657f87657c7cd10f9c50a8af469b7fadab1a3bacb2b1a84abf6bf"} Jul 21 20:14:45.012718 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:45.012685 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmc49" event={"ID":"ee80a941-0e8e-48c3-8bb9-f29e728f895b","Type":"ContainerStarted","Data":"3ac6744c965eac1506236df8ab0d85b099c3f7589d488c7fd5753dc11b43004b"} Jul 21 20:14:45.076717 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:45.076658 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kmc49" podStartSLOduration=9.383889153 podStartE2EDuration="40.07664143s" totalImagesPullingTime="30.692752277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:45.076205955 +0000 UTC m=+39.756332392" watchObservedRunningTime="2026-07-21 20:14:45.07664143 +0000 UTC m=+39.756767867" Jul 21 20:14:46.016607 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:46.016567 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czv2k" event={"ID":"a3190751-b8b6-4693-a0bd-ff461895f8a6","Type":"ContainerStarted","Data":"dd4f22f8dc55f771509de137c5178921273c5c618bc742cc18ebb8e1dd944166"} Jul 21 20:14:46.038486 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:46.038440 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-czv2k" podStartSLOduration=1.720313679 podStartE2EDuration="6.038423603s" totalImagesPullingTime="4.318109924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:14:46.037799836 +0000 UTC m=+40.717926273" watchObservedRunningTime="2026-07-21 20:14:46.038423603 +0000 UTC m=+40.718550056" Jul 21 20:14:47.976470 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:47.976428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:47.976859 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:47.976527 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:47.976859 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:47.976581 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 20:14:47.976859 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:47.976657 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls podName:9bde1aba-deb0-4a77-898b-d3a48bcfca67 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:55.976637597 +0000 UTC m=+50.656764017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls") pod "dns-default-g29mm" (UID: "9bde1aba-deb0-4a77-898b-d3a48bcfca67") : secret "dns-default-metrics-tls" not found Jul 21 20:14:47.976859 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:47.976664 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 20:14:47.976859 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:47.976682 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-664d77bf4-dsmcf: secret "image-registry-tls" not found Jul 21 20:14:47.976859 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:47.976731 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls podName:7fda0385-d801-453a-ab06-bcac6b323a07 nodeName:}" failed. No retries permitted until 2026-07-21 20:14:55.976719199 +0000 UTC m=+50.656845615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls") pod "image-registry-664d77bf4-dsmcf" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07") : secret "image-registry-tls" not found Jul 21 20:14:56.034919 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:56.034881 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:14:56.035396 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:14:56.034936 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:14:56.035396 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:56.035049 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 20:14:56.035396 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:56.035063 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 20:14:56.035396 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:56.035066 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-664d77bf4-dsmcf: secret "image-registry-tls" not found Jul 21 20:14:56.035396 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:56.035123 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls podName:7fda0385-d801-453a-ab06-bcac6b323a07 nodeName:}" failed. No retries permitted until 2026-07-21 20:15:12.03510771 +0000 UTC m=+66.715234125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls") pod "image-registry-664d77bf4-dsmcf" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07") : secret "image-registry-tls" not found Jul 21 20:14:56.035396 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:14:56.035136 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls podName:9bde1aba-deb0-4a77-898b-d3a48bcfca67 nodeName:}" failed. No retries permitted until 2026-07-21 20:15:12.035130613 +0000 UTC m=+66.715257027 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls") pod "dns-default-g29mm" (UID: "9bde1aba-deb0-4a77-898b-d3a48bcfca67") : secret "dns-default-metrics-tls" not found Jul 21 20:15:05.987958 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:05.987917 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nzmdl" Jul 21 20:15:11.548013 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.547964 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:15:11.548013 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.548020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:15:11.551820 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.551802 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 20:15:11.552087 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.552073 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 20:15:11.559172 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:11.559153 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 20:15:11.559229 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:11.559219 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:16:15.559202472 +0000 UTC m=+130.239328887 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : secret "metrics-daemon-secret" not found Jul 21 20:15:11.562775 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.562759 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec48f297-40e1-4b50-a162-d46157ba43da-original-pull-secret\") pod \"global-pull-secret-syncer-zps6q\" (UID: \"ec48f297-40e1-4b50-a162-d46157ba43da\") " pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:15:11.648786 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.648748 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:15:11.651631 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.651614 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 20:15:11.662249 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.662231 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 20:15:11.672566 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.672547 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmxl\" (UniqueName: \"kubernetes.io/projected/55c02ab9-c8c4-451c-867b-9090ac46221c-kube-api-access-7vmxl\") pod \"network-check-target-fkdcg\" (UID: \"55c02ab9-c8c4-451c-867b-9090ac46221c\") " pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:15:11.811135 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.811051 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zps6q" Jul 21 20:15:11.820931 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.820910 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7b7qw\"" Jul 21 20:15:11.828463 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.828441 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:15:11.939512 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.939483 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zps6q"] Jul 21 20:15:11.943463 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:15:11.943438 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec48f297_40e1_4b50_a162_d46157ba43da.slice/crio-5a03244e0a1692405393d79ea24a767e420aa935aca1b4d5d6aaf4667844cb71 WatchSource:0}: Error finding container 5a03244e0a1692405393d79ea24a767e420aa935aca1b4d5d6aaf4667844cb71: Status 404 returned error can't find the container with id 5a03244e0a1692405393d79ea24a767e420aa935aca1b4d5d6aaf4667844cb71 Jul 21 20:15:11.955817 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:11.955796 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fkdcg"] Jul 21 20:15:11.958305 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:15:11.958284 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c02ab9_c8c4_451c_867b_9090ac46221c.slice/crio-8e92c83e2619081076bd9710dcdff578f59fda60eff315e3e740720f5b8cc351 WatchSource:0}: Error finding container 8e92c83e2619081076bd9710dcdff578f59fda60eff315e3e740720f5b8cc351: Status 404 returned error can't find the container with id 8e92c83e2619081076bd9710dcdff578f59fda60eff315e3e740720f5b8cc351 Jul 21 20:15:12.051182 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:12.051146 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:15:12.051327 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:12.051190 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:15:12.051327 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:12.051283 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 20:15:12.051327 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:12.051289 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 20:15:12.051327 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:12.051307 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-664d77bf4-dsmcf: secret "image-registry-tls" not found Jul 21 20:15:12.051468 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:12.051330 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls podName:9bde1aba-deb0-4a77-898b-d3a48bcfca67 nodeName:}" failed. No retries permitted until 2026-07-21 20:15:44.051316293 +0000 UTC m=+98.731442707 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls") pod "dns-default-g29mm" (UID: "9bde1aba-deb0-4a77-898b-d3a48bcfca67") : secret "dns-default-metrics-tls" not found Jul 21 20:15:12.051468 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:12.051352 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls podName:7fda0385-d801-453a-ab06-bcac6b323a07 nodeName:}" failed. No retries permitted until 2026-07-21 20:15:44.051339008 +0000 UTC m=+98.731465423 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls") pod "image-registry-664d77bf4-dsmcf" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07") : secret "image-registry-tls" not found Jul 21 20:15:12.077832 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:12.077736 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fkdcg" event={"ID":"55c02ab9-c8c4-451c-867b-9090ac46221c","Type":"ContainerStarted","Data":"8e92c83e2619081076bd9710dcdff578f59fda60eff315e3e740720f5b8cc351"} Jul 21 20:15:12.078761 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:12.078737 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zps6q" event={"ID":"ec48f297-40e1-4b50-a162-d46157ba43da","Type":"ContainerStarted","Data":"5a03244e0a1692405393d79ea24a767e420aa935aca1b4d5d6aaf4667844cb71"} Jul 21 20:15:17.091365 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:17.091284 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fkdcg" event={"ID":"55c02ab9-c8c4-451c-867b-9090ac46221c","Type":"ContainerStarted","Data":"36c883c6933191fdfb0d5d57920482742bc3fd6b5526621a5eaebd44b062241b"} Jul 21 20:15:17.091365 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:17.091360 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:15:17.092691 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:17.092669 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zps6q" event={"ID":"ec48f297-40e1-4b50-a162-d46157ba43da","Type":"ContainerStarted","Data":"7f2f5fda820e65c6beef5c2bdabe3f9895d73508fbaf896c7eb6ad236b2ee90f"} Jul 21 20:15:17.114519 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:17.114471 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fkdcg" podStartSLOduration=67.270373145 podStartE2EDuration="1m12.114459688s" totalImagesPullingTime="4.844086543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:15:17.114230052 +0000 UTC m=+71.794356508" watchObservedRunningTime="2026-07-21 20:15:17.114459688 +0000 UTC m=+71.794586125" Jul 21 20:15:17.161745 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:17.161696 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zps6q" podStartSLOduration=66.300787758 podStartE2EDuration="1m11.161681646s" totalImagesPullingTime="4.860893888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:15:17.161132637 +0000 UTC m=+71.841259105" watchObservedRunningTime="2026-07-21 20:15:17.161681646 +0000 UTC m=+71.841808119" Jul 21 20:15:44.066414 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:44.066280 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:15:44.066414 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:44.066370 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:15:44.066831 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:44.066452 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 20:15:44.066831 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:44.066481 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 20:15:44.066831 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:44.066494 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-664d77bf4-dsmcf: secret "image-registry-tls" not found Jul 21 20:15:44.066831 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:44.066535 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls podName:9bde1aba-deb0-4a77-898b-d3a48bcfca67 nodeName:}" failed. No retries permitted until 2026-07-21 20:16:48.066516387 +0000 UTC m=+162.746642802 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls") pod "dns-default-g29mm" (UID: "9bde1aba-deb0-4a77-898b-d3a48bcfca67") : secret "dns-default-metrics-tls" not found Jul 21 20:15:44.066831 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:15:44.066552 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls podName:7fda0385-d801-453a-ab06-bcac6b323a07 nodeName:}" failed. No retries permitted until 2026-07-21 20:16:48.066545662 +0000 UTC m=+162.746672077 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls") pod "image-registry-664d77bf4-dsmcf" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07") : secret "image-registry-tls" not found Jul 21 20:15:48.097958 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:15:48.097904 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fkdcg" Jul 21 20:16:15.578018 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:16:15.577977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:16:15.578406 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:15.578124 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 20:16:15.578406 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:15.578199 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:18:17.578183478 +0000 UTC m=+252.258309894 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : secret "metrics-daemon-secret" not found Jul 21 20:16:43.187376 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:43.187331 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" podUID="7fda0385-d801-453a-ab06-bcac6b323a07" Jul 21 20:16:43.217984 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:43.217932 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-g29mm" podUID="9bde1aba-deb0-4a77-898b-d3a48bcfca67" Jul 21 20:16:43.294026 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:16:43.293998 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g29mm" Jul 21 20:16:43.294178 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:16:43.293998 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:16:44.823143 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:44.823101 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:16:48.095435 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:16:48.095397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:16:48.095897 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:16:48.095486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:16:48.095897 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:48.095548 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 20:16:48.095897 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:48.095599 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 20:16:48.095897 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:48.095617 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-664d77bf4-dsmcf: secret "image-registry-tls" not found Jul 21 20:16:48.095897 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:48.095620 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls podName:9bde1aba-deb0-4a77-898b-d3a48bcfca67 nodeName:}" failed. No retries permitted until 2026-07-21 20:18:50.095599808 +0000 UTC m=+284.775726222 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls") pod "dns-default-g29mm" (UID: "9bde1aba-deb0-4a77-898b-d3a48bcfca67") : secret "dns-default-metrics-tls" not found Jul 21 20:16:48.095897 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:16:48.095660 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls podName:7fda0385-d801-453a-ab06-bcac6b323a07 nodeName:}" failed. No retries permitted until 2026-07-21 20:18:50.095648369 +0000 UTC m=+284.775774784 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls") pod "image-registry-664d77bf4-dsmcf" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07") : secret "image-registry-tls" not found Jul 21 20:16:58.799139 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:16:58.799057 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:18:15.207665 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.207633 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-95dqt"] Jul 21 20:18:15.210431 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.210415 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.212373 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.212348 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7"] Jul 21 20:18:15.214899 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.214883 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b"] Jul 21 20:18:15.215009 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.214993 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:15.215069 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.215050 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 20:18:15.215162 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.215118 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 20:18:15.215894 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.215873 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 20:18:15.216370 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.216253 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 20:18:15.216648 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.216632 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pgdv4\"" Jul 21 20:18:15.217500 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.217481 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v"] Jul 21 20:18:15.217628 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.217609 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.218320 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.218190 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 21 20:18:15.218560 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.218545 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4mdt7\"" Jul 21 20:18:15.219001 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.218987 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 21 20:18:15.220353 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.220339 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.220857 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.220839 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-92bv8\"" Jul 21 20:18:15.221780 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.221757 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 21 20:18:15.221876 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.221793 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 21 20:18:15.221876 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.221828 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 20:18:15.221876 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.221843 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 20:18:15.222367 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.222352 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bxnrq\"" Jul 21 20:18:15.222441 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.222374 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 20:18:15.222441 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.222421 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 20:18:15.222599 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.222585 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 20:18:15.223110 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.223088 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 20:18:15.225410 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.225391 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 20:18:15.229708 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.229689 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-95dqt"] Jul 21 20:18:15.230575 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.230546 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7"] Jul 21 20:18:15.231750 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.231693 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b"] Jul 21 20:18:15.240432 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.240410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v"] Jul 21 20:18:15.303446 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303413 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4"] Jul 21 20:18:15.303617 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303580 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwq7\" (UniqueName: \"kubernetes.io/projected/7de5f4f5-4560-4fd3-93b3-ebb043fe600f-kube-api-access-5kwq7\") pod \"service-ca-operator-5bcbc94c78-f4b7v\" (UID: \"7de5f4f5-4560-4fd3-93b3-ebb043fe600f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.303676 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303623 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7de5f4f5-4560-4fd3-93b3-ebb043fe600f-config\") pod \"service-ca-operator-5bcbc94c78-f4b7v\" (UID: \"7de5f4f5-4560-4fd3-93b3-ebb043fe600f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.303676 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303648 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7de5f4f5-4560-4fd3-93b3-ebb043fe600f-serving-cert\") pod \"service-ca-operator-5bcbc94c78-f4b7v\" (UID: \"7de5f4f5-4560-4fd3-93b3-ebb043fe600f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.303676 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhczh\" (UniqueName: \"kubernetes.io/projected/4aec0aca-f346-46fb-8392-24e10c582da2-kube-api-access-jhczh\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.303822 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303688 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:15.303822 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303704 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.303822 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303754 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vn68\" (UniqueName: \"kubernetes.io/projected/0af8ce4b-1621-4236-87e4-3705792ab31b-kube-api-access-8vn68\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.303822 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303790 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aca-f346-46fb-8392-24e10c582da2-tmp\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.303822 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303817 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4aec0aca-f346-46fb-8392-24e10c582da2-serving-cert\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.304089 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303833 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0af8ce4b-1621-4236-87e4-3705792ab31b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.304089 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303857 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:15.304089 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303874 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aec0aca-f346-46fb-8392-24e10c582da2-service-ca-bundle\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.304089 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303902 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aec0aca-f346-46fb-8392-24e10c582da2-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.304089 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.303922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aca-f346-46fb-8392-24e10c582da2-snapshots\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.306590 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.306572 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4" Jul 21 20:18:15.309296 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.309278 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 21 20:18:15.309393 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.309279 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-5xtxz\"" Jul 21 20:18:15.309393 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.309320 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 21 20:18:15.316769 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.316749 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq"] Jul 21 20:18:15.319747 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.319729 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4"] Jul 21 20:18:15.319837 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.319823 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq" Jul 21 20:18:15.321962 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.321933 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xnc9h\"" Jul 21 20:18:15.352291 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.352248 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq"] Jul 21 20:18:15.404752 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.404712 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:15.404752 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.404749 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aec0aca-f346-46fb-8392-24e10c582da2-service-ca-bundle\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.405012 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.404773 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aec0aca-f346-46fb-8392-24e10c582da2-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.405012 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:15.404872 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 20:18:15.405012 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.404906 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4znh\" (UniqueName: \"kubernetes.io/projected/0357583f-0e56-4bd8-8b7f-a5f47427bd84-kube-api-access-q4znh\") pod \"network-check-source-7bc5b6bf85-cvsjq\" (UID: \"0357583f-0e56-4bd8-8b7f-a5f47427bd84\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq" Jul 21 20:18:15.405012 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:15.404965 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert podName:c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8 nodeName:}" failed. No retries permitted until 2026-07-21 20:18:15.904924589 +0000 UTC m=+250.585051008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-vddm7" (UID: "c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8") : secret "networking-console-plugin-cert" not found Jul 21 20:18:15.405179 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aca-f346-46fb-8392-24e10c582da2-snapshots\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.405179 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405105 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwq7\" (UniqueName: \"kubernetes.io/projected/7de5f4f5-4560-4fd3-93b3-ebb043fe600f-kube-api-access-5kwq7\") pod \"service-ca-operator-5bcbc94c78-f4b7v\" (UID: \"7de5f4f5-4560-4fd3-93b3-ebb043fe600f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.405179 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kbt\" (UniqueName: \"kubernetes.io/projected/9c0b6228-cec1-498a-8bef-bd0087d6d153-kube-api-access-t7kbt\") pod \"volume-data-source-validator-78cf4c6f6-hbsv4\" (UID: \"9c0b6228-cec1-498a-8bef-bd0087d6d153\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4" Jul 21 20:18:15.405312 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405199 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7de5f4f5-4560-4fd3-93b3-ebb043fe600f-config\") pod \"service-ca-operator-5bcbc94c78-f4b7v\" (UID: \"7de5f4f5-4560-4fd3-93b3-ebb043fe600f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.405312 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405226 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7de5f4f5-4560-4fd3-93b3-ebb043fe600f-serving-cert\") pod \"service-ca-operator-5bcbc94c78-f4b7v\" (UID: \"7de5f4f5-4560-4fd3-93b3-ebb043fe600f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.405312 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405265 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhczh\" (UniqueName: \"kubernetes.io/projected/4aec0aca-f346-46fb-8392-24e10c582da2-kube-api-access-jhczh\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.405312 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405296 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:15.405681 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.405681 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vn68\" (UniqueName: \"kubernetes.io/projected/0af8ce4b-1621-4236-87e4-3705792ab31b-kube-api-access-8vn68\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.405681 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405372 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aca-f346-46fb-8392-24e10c582da2-tmp\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.405681 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405395 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4aec0aca-f346-46fb-8392-24e10c582da2-serving-cert\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.405681 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405415 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0af8ce4b-1621-4236-87e4-3705792ab31b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.405681 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405486 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aec0aca-f346-46fb-8392-24e10c582da2-service-ca-bundle\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.405681 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:15.405515 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:15.405681 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:15.405575 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls podName:0af8ce4b-1621-4236-87e4-3705792ab31b nodeName:}" failed. No retries permitted until 2026-07-21 20:18:15.905557867 +0000 UTC m=+250.585684292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-r5j5b" (UID: "0af8ce4b-1621-4236-87e4-3705792ab31b") : secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:15.406005 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.405686 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7de5f4f5-4560-4fd3-93b3-ebb043fe600f-config\") pod \"service-ca-operator-5bcbc94c78-f4b7v\" (UID: \"7de5f4f5-4560-4fd3-93b3-ebb043fe600f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.406083 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.406062 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aca-f346-46fb-8392-24e10c582da2-tmp\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.406142 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.406121 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4aec0aca-f346-46fb-8392-24e10c582da2-snapshots\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.406553 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.406329 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4aec0aca-f346-46fb-8392-24e10c582da2-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.406670 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.406638 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:15.406670 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.406638 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0af8ce4b-1621-4236-87e4-3705792ab31b-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.410366 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.409023 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7de5f4f5-4560-4fd3-93b3-ebb043fe600f-serving-cert\") pod \"service-ca-operator-5bcbc94c78-f4b7v\" (UID: \"7de5f4f5-4560-4fd3-93b3-ebb043fe600f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.410366 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.409086 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4aec0aca-f346-46fb-8392-24e10c582da2-serving-cert\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.429072 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.429050 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhczh\" (UniqueName: \"kubernetes.io/projected/4aec0aca-f346-46fb-8392-24e10c582da2-kube-api-access-jhczh\") pod \"insights-operator-555d9d7885-95dqt\" (UID: \"4aec0aca-f346-46fb-8392-24e10c582da2\") " pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.450163 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.450136 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwq7\" (UniqueName: \"kubernetes.io/projected/7de5f4f5-4560-4fd3-93b3-ebb043fe600f-kube-api-access-5kwq7\") pod \"service-ca-operator-5bcbc94c78-f4b7v\" (UID: \"7de5f4f5-4560-4fd3-93b3-ebb043fe600f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.452230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.452208 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vn68\" (UniqueName: \"kubernetes.io/projected/0af8ce4b-1621-4236-87e4-3705792ab31b-kube-api-access-8vn68\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.473691 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.473605 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5"] Jul 21 20:18:15.477949 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.477920 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.480220 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.480201 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 20:18:15.480382 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.480356 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 20:18:15.480490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.480409 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-6htc5\"" Jul 21 20:18:15.480490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.480360 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 20:18:15.480490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.480447 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 20:18:15.488053 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.488032 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5"] Jul 21 20:18:15.506017 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.505962 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b28e11a-e4ce-4b63-966d-60864726d9fc-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-66qg5\" (UID: \"3b28e11a-e4ce-4b63-966d-60864726d9fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.506153 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.506046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kbt\" (UniqueName: \"kubernetes.io/projected/9c0b6228-cec1-498a-8bef-bd0087d6d153-kube-api-access-t7kbt\") pod \"volume-data-source-validator-78cf4c6f6-hbsv4\" (UID: \"9c0b6228-cec1-498a-8bef-bd0087d6d153\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4" Jul 21 20:18:15.506230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.506153 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b28e11a-e4ce-4b63-966d-60864726d9fc-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-66qg5\" (UID: \"3b28e11a-e4ce-4b63-966d-60864726d9fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.506230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.506219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqp7\" (UniqueName: \"kubernetes.io/projected/3b28e11a-e4ce-4b63-966d-60864726d9fc-kube-api-access-6zqp7\") pod \"kube-storage-version-migrator-operator-7658d9b749-66qg5\" (UID: \"3b28e11a-e4ce-4b63-966d-60864726d9fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.506485 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.506271 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4znh\" (UniqueName: \"kubernetes.io/projected/0357583f-0e56-4bd8-8b7f-a5f47427bd84-kube-api-access-q4znh\") pod \"network-check-source-7bc5b6bf85-cvsjq\" (UID: \"0357583f-0e56-4bd8-8b7f-a5f47427bd84\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq" Jul 21 20:18:15.514112 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.514082 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4znh\" (UniqueName: \"kubernetes.io/projected/0357583f-0e56-4bd8-8b7f-a5f47427bd84-kube-api-access-q4znh\") pod \"network-check-source-7bc5b6bf85-cvsjq\" (UID: \"0357583f-0e56-4bd8-8b7f-a5f47427bd84\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq" Jul 21 20:18:15.514112 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.514092 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kbt\" (UniqueName: \"kubernetes.io/projected/9c0b6228-cec1-498a-8bef-bd0087d6d153-kube-api-access-t7kbt\") pod \"volume-data-source-validator-78cf4c6f6-hbsv4\" (UID: \"9c0b6228-cec1-498a-8bef-bd0087d6d153\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4" Jul 21 20:18:15.521859 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.521844 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-95dqt" Jul 21 20:18:15.541912 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.541892 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" Jul 21 20:18:15.607753 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.607601 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b28e11a-e4ce-4b63-966d-60864726d9fc-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-66qg5\" (UID: \"3b28e11a-e4ce-4b63-966d-60864726d9fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.607753 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.607705 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b28e11a-e4ce-4b63-966d-60864726d9fc-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-66qg5\" (UID: \"3b28e11a-e4ce-4b63-966d-60864726d9fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.607887 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.607770 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqp7\" (UniqueName: \"kubernetes.io/projected/3b28e11a-e4ce-4b63-966d-60864726d9fc-kube-api-access-6zqp7\") pod \"kube-storage-version-migrator-operator-7658d9b749-66qg5\" (UID: \"3b28e11a-e4ce-4b63-966d-60864726d9fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.609116 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.609061 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b28e11a-e4ce-4b63-966d-60864726d9fc-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-66qg5\" (UID: \"3b28e11a-e4ce-4b63-966d-60864726d9fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.610135 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.610115 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b28e11a-e4ce-4b63-966d-60864726d9fc-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-66qg5\" (UID: \"3b28e11a-e4ce-4b63-966d-60864726d9fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.615149 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.615129 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4" Jul 21 20:18:15.622761 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.622707 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqp7\" (UniqueName: \"kubernetes.io/projected/3b28e11a-e4ce-4b63-966d-60864726d9fc-kube-api-access-6zqp7\") pod \"kube-storage-version-migrator-operator-7658d9b749-66qg5\" (UID: \"3b28e11a-e4ce-4b63-966d-60864726d9fc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.627217 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.627193 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq" Jul 21 20:18:15.658081 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.657864 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-95dqt"] Jul 21 20:18:15.661611 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:15.661572 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aec0aca_f346_46fb_8392_24e10c582da2.slice/crio-4fea82a7c9fa4e5aee0185a8b20b3501413f4e8bf08d6c39aae679206188933a WatchSource:0}: Error finding container 4fea82a7c9fa4e5aee0185a8b20b3501413f4e8bf08d6c39aae679206188933a: Status 404 returned error can't find the container with id 4fea82a7c9fa4e5aee0185a8b20b3501413f4e8bf08d6c39aae679206188933a Jul 21 20:18:15.674435 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.674347 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v"] Jul 21 20:18:15.678295 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:15.678251 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de5f4f5_4560_4fd3_93b3_ebb043fe600f.slice/crio-b05f24c76e992aebedc9dae495d2141f4e42dcb1dc157b7b4424b59e1bbb41d0 WatchSource:0}: Error finding container b05f24c76e992aebedc9dae495d2141f4e42dcb1dc157b7b4424b59e1bbb41d0: Status 404 returned error can't find the container with id b05f24c76e992aebedc9dae495d2141f4e42dcb1dc157b7b4424b59e1bbb41d0 Jul 21 20:18:15.754567 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.754536 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4"] Jul 21 20:18:15.757473 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:15.757437 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0b6228_cec1_498a_8bef_bd0087d6d153.slice/crio-b78e76a63079b0886b3e4131b8056ee943808fd26f12a77b7458b5f0958dfd5d WatchSource:0}: Error finding container b78e76a63079b0886b3e4131b8056ee943808fd26f12a77b7458b5f0958dfd5d: Status 404 returned error can't find the container with id b78e76a63079b0886b3e4131b8056ee943808fd26f12a77b7458b5f0958dfd5d Jul 21 20:18:15.771001 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.770976 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq"] Jul 21 20:18:15.774089 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:15.774066 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0357583f_0e56_4bd8_8b7f_a5f47427bd84.slice/crio-7f90a6698d8be9761de832af6c3cea641ef0c55a1e05592f4840856a7644666f WatchSource:0}: Error finding container 7f90a6698d8be9761de832af6c3cea641ef0c55a1e05592f4840856a7644666f: Status 404 returned error can't find the container with id 7f90a6698d8be9761de832af6c3cea641ef0c55a1e05592f4840856a7644666f Jul 21 20:18:15.787566 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.787546 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" Jul 21 20:18:15.910714 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.910686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:15.910856 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.910727 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:15.910856 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:15.910821 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 20:18:15.910856 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:15.910824 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:15.911002 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:15.910871 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert podName:c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8 nodeName:}" failed. No retries permitted until 2026-07-21 20:18:16.910857336 +0000 UTC m=+251.590983751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-vddm7" (UID: "c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8") : secret "networking-console-plugin-cert" not found Jul 21 20:18:15.911002 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:15.910883 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls podName:0af8ce4b-1621-4236-87e4-3705792ab31b nodeName:}" failed. No retries permitted until 2026-07-21 20:18:16.910877861 +0000 UTC m=+251.591004276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-r5j5b" (UID: "0af8ce4b-1621-4236-87e4-3705792ab31b") : secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:15.918477 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:15.918448 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5"] Jul 21 20:18:15.923830 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:15.923809 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b28e11a_e4ce_4b63_966d_60864726d9fc.slice/crio-36bb4a45c5b78dc17fa360d3dfc88ae412f9bac558c91f030c6055a6ff7b9543 WatchSource:0}: Error finding container 36bb4a45c5b78dc17fa360d3dfc88ae412f9bac558c91f030c6055a6ff7b9543: Status 404 returned error can't find the container with id 36bb4a45c5b78dc17fa360d3dfc88ae412f9bac558c91f030c6055a6ff7b9543 Jul 21 20:18:16.510973 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:16.510869 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" event={"ID":"3b28e11a-e4ce-4b63-966d-60864726d9fc","Type":"ContainerStarted","Data":"36bb4a45c5b78dc17fa360d3dfc88ae412f9bac558c91f030c6055a6ff7b9543"} Jul 21 20:18:16.514008 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:16.513358 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq" event={"ID":"0357583f-0e56-4bd8-8b7f-a5f47427bd84","Type":"ContainerStarted","Data":"4128eade23a369e97294b279c1d6c01ab5ae932369094ea9c340ec956d8488a7"} Jul 21 20:18:16.514008 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:16.513403 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq" event={"ID":"0357583f-0e56-4bd8-8b7f-a5f47427bd84","Type":"ContainerStarted","Data":"7f90a6698d8be9761de832af6c3cea641ef0c55a1e05592f4840856a7644666f"} Jul 21 20:18:16.515995 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:16.515912 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4" event={"ID":"9c0b6228-cec1-498a-8bef-bd0087d6d153","Type":"ContainerStarted","Data":"b78e76a63079b0886b3e4131b8056ee943808fd26f12a77b7458b5f0958dfd5d"} Jul 21 20:18:16.518795 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:16.518770 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" event={"ID":"7de5f4f5-4560-4fd3-93b3-ebb043fe600f","Type":"ContainerStarted","Data":"b05f24c76e992aebedc9dae495d2141f4e42dcb1dc157b7b4424b59e1bbb41d0"} Jul 21 20:18:16.520383 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:16.520328 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-95dqt" event={"ID":"4aec0aca-f346-46fb-8392-24e10c582da2","Type":"ContainerStarted","Data":"4fea82a7c9fa4e5aee0185a8b20b3501413f4e8bf08d6c39aae679206188933a"} Jul 21 20:18:16.533382 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:16.532891 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-cvsjq" podStartSLOduration=1.532872792 podStartE2EDuration="1.532872792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:16.531788076 +0000 UTC m=+251.211914511" watchObservedRunningTime="2026-07-21 20:18:16.532872792 +0000 UTC m=+251.212999229" Jul 21 20:18:16.920476 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:16.919691 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:16.920476 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:16.919759 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:16.920476 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:16.919912 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 20:18:16.920476 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:16.919996 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert podName:c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8 nodeName:}" failed. No retries permitted until 2026-07-21 20:18:18.919977146 +0000 UTC m=+253.600103567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-vddm7" (UID: "c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8") : secret "networking-console-plugin-cert" not found Jul 21 20:18:16.920476 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:16.920384 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:16.920476 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:16.920434 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls podName:0af8ce4b-1621-4236-87e4-3705792ab31b nodeName:}" failed. No retries permitted until 2026-07-21 20:18:18.920418937 +0000 UTC m=+253.600545358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-r5j5b" (UID: "0af8ce4b-1621-4236-87e4-3705792ab31b") : secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:17.525787 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:17.525697 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4" event={"ID":"9c0b6228-cec1-498a-8bef-bd0087d6d153","Type":"ContainerStarted","Data":"ced999ee8507c22332e5c2ad54d1beb5026771823dedab4976c63b221a1d5076"} Jul 21 20:18:17.626591 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:17.626514 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:18:17.626761 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:17.626708 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 20:18:17.626810 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:17.626778 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs podName:011ea63a-4a29-4f3e-9710-4b9b299ba69c nodeName:}" failed. No retries permitted until 2026-07-21 20:20:19.626757166 +0000 UTC m=+374.306883585 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs") pod "network-metrics-daemon-f57lq" (UID: "011ea63a-4a29-4f3e-9710-4b9b299ba69c") : secret "metrics-daemon-secret" not found Jul 21 20:18:18.936380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:18.936321 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:18.936855 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:18.936394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:18.936855 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:18.936518 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 20:18:18.936855 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:18.936583 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert podName:c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8 nodeName:}" failed. No retries permitted until 2026-07-21 20:18:22.936565424 +0000 UTC m=+257.616691840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-vddm7" (UID: "c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8") : secret "networking-console-plugin-cert" not found Jul 21 20:18:18.936855 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:18.936670 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:18.936855 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:18.936694 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls podName:0af8ce4b-1621-4236-87e4-3705792ab31b nodeName:}" failed. No retries permitted until 2026-07-21 20:18:22.936686965 +0000 UTC m=+257.616813380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-r5j5b" (UID: "0af8ce4b-1621-4236-87e4-3705792ab31b") : secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:19.534745 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:19.534644 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" event={"ID":"3b28e11a-e4ce-4b63-966d-60864726d9fc","Type":"ContainerStarted","Data":"afcb0f271e14fb5b2860b346361a5da9d4a7ec94a6984a803e231e62c03f53a8"} Jul 21 20:18:19.536329 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:19.536296 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" event={"ID":"7de5f4f5-4560-4fd3-93b3-ebb043fe600f","Type":"ContainerStarted","Data":"1108d7f8d24233f23782659586bdfc1b6417d3c47c902e91f23d1c1427637877"} Jul 21 20:18:19.537969 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:19.537929 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-95dqt" event={"ID":"4aec0aca-f346-46fb-8392-24e10c582da2","Type":"ContainerStarted","Data":"7e8b7569d69b2b1badc03e008c8209a84a7b6a18c2e7c8ab63573186c6bd18f5"} Jul 21 20:18:19.553064 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:19.553008 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" podStartSLOduration=1.331457799 podStartE2EDuration="4.552990294s" totalImagesPullingTime="3.221532495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:19.551821174 +0000 UTC m=+254.231947613" watchObservedRunningTime="2026-07-21 20:18:19.552990294 +0000 UTC m=+254.233116732" Jul 21 20:18:19.553821 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:19.553791 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-hbsv4" podStartSLOduration=3.044999866 podStartE2EDuration="4.553780577s" totalImagesPullingTime="1.508780711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:17.544238624 +0000 UTC m=+252.224365090" watchObservedRunningTime="2026-07-21 20:18:19.553780577 +0000 UTC m=+254.233907015" Jul 21 20:18:19.572215 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:19.572147 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-95dqt" podStartSLOduration=1.083381968 podStartE2EDuration="4.572131984s" totalImagesPullingTime="3.488750016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:19.571044599 +0000 UTC m=+254.251171038" watchObservedRunningTime="2026-07-21 20:18:19.572131984 +0000 UTC m=+254.252258421" Jul 21 20:18:19.591438 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:19.591346 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" podStartSLOduration=1.121589788 podStartE2EDuration="4.591330346s" totalImagesPullingTime="3.469740558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:19.590205006 +0000 UTC m=+254.270331444" watchObservedRunningTime="2026-07-21 20:18:19.591330346 +0000 UTC m=+254.271456786" Jul 21 20:18:20.464813 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.464762 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-57t7d"] Jul 21 20:18:20.469867 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.469840 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.473157 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.473129 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 20:18:20.473309 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.473223 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7vzqq\"" Jul 21 20:18:20.474028 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.474013 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 20:18:20.484266 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.484237 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-57t7d"] Jul 21 20:18:20.551657 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.551628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.551835 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.551706 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48f4c58e-85c1-435a-bd62-30c0950be9ba-crio-socket\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.551835 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.551753 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48f4c58e-85c1-435a-bd62-30c0950be9ba-data-volume\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.551835 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.551775 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.552012 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.551834 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqr2\" (UniqueName: \"kubernetes.io/projected/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-api-access-qlqr2\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.652629 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.652590 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqr2\" (UniqueName: \"kubernetes.io/projected/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-api-access-qlqr2\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.652910 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.652888 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.653080 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:20.653003 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 20:18:20.653080 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.653046 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48f4c58e-85c1-435a-bd62-30c0950be9ba-crio-socket\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.653080 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:20.653077 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:21.153057736 +0000 UTC m=+255.833184160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : configmap "kube-rbac-proxy" not found Jul 21 20:18:20.653277 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.653166 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48f4c58e-85c1-435a-bd62-30c0950be9ba-data-volume\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.653277 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.653196 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.653277 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.653240 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48f4c58e-85c1-435a-bd62-30c0950be9ba-crio-socket\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.653461 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:20.653349 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 20:18:20.653461 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:20.653408 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:21.153391102 +0000 UTC m=+255.833517523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : secret "insights-runtime-extractor-tls" not found Jul 21 20:18:20.653587 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.653567 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48f4c58e-85c1-435a-bd62-30c0950be9ba-data-volume\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.663776 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.663748 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqr2\" (UniqueName: \"kubernetes.io/projected/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-api-access-qlqr2\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:20.997364 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:20.997333 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n"] Jul 21 20:18:21.000637 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.000619 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n" Jul 21 20:18:21.002996 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.002976 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 21 20:18:21.003145 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.003026 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 21 20:18:21.003348 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.003329 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hzrk6\"" Jul 21 20:18:21.009365 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.009343 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n"] Jul 21 20:18:21.057170 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.057144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slsm8\" (UniqueName: \"kubernetes.io/projected/afc7da83-8a9f-4118-93a6-d55193041472-kube-api-access-slsm8\") pod \"migrator-7cc9c7ffd7-qwp4n\" (UID: \"afc7da83-8a9f-4118-93a6-d55193041472\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n" Jul 21 20:18:21.157528 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.157492 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:21.157528 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.157532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slsm8\" (UniqueName: \"kubernetes.io/projected/afc7da83-8a9f-4118-93a6-d55193041472-kube-api-access-slsm8\") pod \"migrator-7cc9c7ffd7-qwp4n\" (UID: \"afc7da83-8a9f-4118-93a6-d55193041472\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n" Jul 21 20:18:21.157757 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:21.157639 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 20:18:21.157757 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.157681 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:21.157757 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:21.157711 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:22.157692163 +0000 UTC m=+256.837818587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : configmap "kube-rbac-proxy" not found Jul 21 20:18:21.157757 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:21.157755 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 20:18:21.157990 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:21.157797 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:22.157785301 +0000 UTC m=+256.837911715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : secret "insights-runtime-extractor-tls" not found Jul 21 20:18:21.167411 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.167384 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slsm8\" (UniqueName: \"kubernetes.io/projected/afc7da83-8a9f-4118-93a6-d55193041472-kube-api-access-slsm8\") pod \"migrator-7cc9c7ffd7-qwp4n\" (UID: \"afc7da83-8a9f-4118-93a6-d55193041472\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n" Jul 21 20:18:21.310261 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.310179 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n" Jul 21 20:18:21.431403 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.431372 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n"] Jul 21 20:18:21.434702 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:21.434678 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc7da83_8a9f_4118_93a6_d55193041472.slice/crio-c8c00960680c3a599f96caaf36ae689b71a14a4e88594cfda10a3dd3182ef1d6 WatchSource:0}: Error finding container c8c00960680c3a599f96caaf36ae689b71a14a4e88594cfda10a3dd3182ef1d6: Status 404 returned error can't find the container with id c8c00960680c3a599f96caaf36ae689b71a14a4e88594cfda10a3dd3182ef1d6 Jul 21 20:18:21.546051 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:21.546015 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n" event={"ID":"afc7da83-8a9f-4118-93a6-d55193041472","Type":"ContainerStarted","Data":"c8c00960680c3a599f96caaf36ae689b71a14a4e88594cfda10a3dd3182ef1d6"} Jul 21 20:18:22.165511 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:22.165480 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:22.165709 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:22.165543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:22.165709 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:22.165608 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 20:18:22.165709 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:22.165665 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 20:18:22.165709 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:22.165677 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:24.165662815 +0000 UTC m=+258.845789231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : configmap "kube-rbac-proxy" not found Jul 21 20:18:22.165865 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:22.165715 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:24.165701638 +0000 UTC m=+258.845828052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : secret "insights-runtime-extractor-tls" not found Jul 21 20:18:22.258882 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:22.258855 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfpqq_6e05dd41-bfb8-4bd8-bfe4-955dce5591f4/dns-node-resolver/0.log" Jul 21 20:18:22.970752 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:22.970711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:22.971246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:22.970796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:22.971246 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:22.970891 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:22.971246 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:22.970916 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 20:18:22.971246 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:22.970992 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls podName:0af8ce4b-1621-4236-87e4-3705792ab31b nodeName:}" failed. No retries permitted until 2026-07-21 20:18:30.970970795 +0000 UTC m=+265.651097223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-r5j5b" (UID: "0af8ce4b-1621-4236-87e4-3705792ab31b") : secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:22.971246 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:22.971013 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert podName:c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8 nodeName:}" failed. No retries permitted until 2026-07-21 20:18:30.971003633 +0000 UTC m=+265.651130055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-vddm7" (UID: "c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8") : secret "networking-console-plugin-cert" not found Jul 21 20:18:23.067595 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.067568 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-vsghl"] Jul 21 20:18:23.071685 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.071663 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.074166 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.074144 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 21 20:18:23.074263 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.074149 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 21 20:18:23.074809 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.074791 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7cz9n\"" Jul 21 20:18:23.074912 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.074793 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 21 20:18:23.074912 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.074839 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 21 20:18:23.081124 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.081105 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-vsghl"] Jul 21 20:18:23.172741 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.172720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b3c6e842-cafe-4bce-856c-051b7215205f-signing-cabundle\") pod \"service-ca-58598695f7-vsghl\" (UID: \"b3c6e842-cafe-4bce-856c-051b7215205f\") " pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.172844 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.172755 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrk8d\" (UniqueName: \"kubernetes.io/projected/b3c6e842-cafe-4bce-856c-051b7215205f-kube-api-access-zrk8d\") pod \"service-ca-58598695f7-vsghl\" (UID: \"b3c6e842-cafe-4bce-856c-051b7215205f\") " pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.172901 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.172836 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b3c6e842-cafe-4bce-856c-051b7215205f-signing-key\") pod \"service-ca-58598695f7-vsghl\" (UID: \"b3c6e842-cafe-4bce-856c-051b7215205f\") " pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.261255 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.261189 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4dc2n_32f0ea72-8c4a-4787-bb7f-b6aa0991eef2/node-ca/0.log" Jul 21 20:18:23.273600 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.273573 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b3c6e842-cafe-4bce-856c-051b7215205f-signing-cabundle\") pod \"service-ca-58598695f7-vsghl\" (UID: \"b3c6e842-cafe-4bce-856c-051b7215205f\") " pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.273600 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.273601 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrk8d\" (UniqueName: \"kubernetes.io/projected/b3c6e842-cafe-4bce-856c-051b7215205f-kube-api-access-zrk8d\") pod \"service-ca-58598695f7-vsghl\" (UID: \"b3c6e842-cafe-4bce-856c-051b7215205f\") " pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.273793 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.273650 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b3c6e842-cafe-4bce-856c-051b7215205f-signing-key\") pod \"service-ca-58598695f7-vsghl\" (UID: \"b3c6e842-cafe-4bce-856c-051b7215205f\") " pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.274272 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.274246 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b3c6e842-cafe-4bce-856c-051b7215205f-signing-cabundle\") pod \"service-ca-58598695f7-vsghl\" (UID: \"b3c6e842-cafe-4bce-856c-051b7215205f\") " pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.275996 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.275977 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b3c6e842-cafe-4bce-856c-051b7215205f-signing-key\") pod \"service-ca-58598695f7-vsghl\" (UID: \"b3c6e842-cafe-4bce-856c-051b7215205f\") " pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.283183 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.283155 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrk8d\" (UniqueName: \"kubernetes.io/projected/b3c6e842-cafe-4bce-856c-051b7215205f-kube-api-access-zrk8d\") pod \"service-ca-58598695f7-vsghl\" (UID: \"b3c6e842-cafe-4bce-856c-051b7215205f\") " pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.383239 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.383206 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-vsghl" Jul 21 20:18:23.506441 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.506354 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-vsghl"] Jul 21 20:18:23.510268 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:23.510243 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c6e842_cafe_4bce_856c_051b7215205f.slice/crio-1c397a2b0b283e7c3fa84bbd95b4c94e4aed495352f33dfa869644549ba6e0d8 WatchSource:0}: Error finding container 1c397a2b0b283e7c3fa84bbd95b4c94e4aed495352f33dfa869644549ba6e0d8: Status 404 returned error can't find the container with id 1c397a2b0b283e7c3fa84bbd95b4c94e4aed495352f33dfa869644549ba6e0d8 Jul 21 20:18:23.553824 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.553794 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n" event={"ID":"afc7da83-8a9f-4118-93a6-d55193041472","Type":"ContainerStarted","Data":"9a47dd715e9023ef7d6f42812d288a27ab04a78ad96962e75215665ca99359b8"} Jul 21 20:18:23.553961 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.553856 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n" event={"ID":"afc7da83-8a9f-4118-93a6-d55193041472","Type":"ContainerStarted","Data":"6e5ef74ead599359ef2ec62715242c412e61808f55502e3edea8f37801247423"} Jul 21 20:18:23.554808 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.554789 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-vsghl" event={"ID":"b3c6e842-cafe-4bce-856c-051b7215205f","Type":"ContainerStarted","Data":"1c397a2b0b283e7c3fa84bbd95b4c94e4aed495352f33dfa869644549ba6e0d8"} Jul 21 20:18:23.572409 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:23.572369 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-qwp4n" podStartSLOduration=1.967509629 podStartE2EDuration="3.572354252s" totalImagesPullingTime="1.604844623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:23.571624782 +0000 UTC m=+258.251751237" watchObservedRunningTime="2026-07-21 20:18:23.572354252 +0000 UTC m=+258.252480689" Jul 21 20:18:24.063432 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:24.063407 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-czv2k_a3190751-b8b6-4693-a0bd-ff461895f8a6/serve-healthcheck-canary/0.log" Jul 21 20:18:24.181429 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:24.181384 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:24.181623 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:24.181465 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:24.181623 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:24.181538 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 20:18:24.181623 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:24.181562 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 20:18:24.181623 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:24.181618 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:28.181595285 +0000 UTC m=+262.861721705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : configmap "kube-rbac-proxy" not found Jul 21 20:18:24.181900 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:24.181647 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:28.181636722 +0000 UTC m=+262.861763144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : secret "insights-runtime-extractor-tls" not found Jul 21 20:18:24.559790 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:24.559758 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-vsghl" event={"ID":"b3c6e842-cafe-4bce-856c-051b7215205f","Type":"ContainerStarted","Data":"1b92c11d95a7d2e6f535832687b219cccb168f56bd31fe58bd788a54defb37eb"} Jul 21 20:18:28.212619 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:28.212541 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:28.212999 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:28.212708 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 20:18:28.212999 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:28.212718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:28.212999 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:28.212797 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:36.212782197 +0000 UTC m=+270.892908611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : secret "insights-runtime-extractor-tls" not found Jul 21 20:18:28.212999 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:28.212823 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 20:18:28.212999 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:28.212867 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm podName:48f4c58e-85c1-435a-bd62-30c0950be9ba nodeName:}" failed. No retries permitted until 2026-07-21 20:18:36.212856382 +0000 UTC m=+270.892982797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm") pod "insights-runtime-extractor-57t7d" (UID: "48f4c58e-85c1-435a-bd62-30c0950be9ba") : configmap "kube-rbac-proxy" not found Jul 21 20:18:31.039130 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:31.039089 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:31.039564 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:31.039152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:31.039564 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:31.039238 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:31.039564 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:31.039250 2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 21 20:18:31.039564 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:31.039300 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert podName:c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8 nodeName:}" failed. No retries permitted until 2026-07-21 20:18:47.039287708 +0000 UTC m=+281.719414123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-vddm7" (UID: "c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8") : secret "networking-console-plugin-cert" not found Jul 21 20:18:31.039564 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:31.039313 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls podName:0af8ce4b-1621-4236-87e4-3705792ab31b nodeName:}" failed. No retries permitted until 2026-07-21 20:18:47.039307257 +0000 UTC m=+281.719433672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-r5j5b" (UID: "0af8ce4b-1621-4236-87e4-3705792ab31b") : secret "cluster-monitoring-operator-tls" not found Jul 21 20:18:36.285112 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:36.285076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:36.285488 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:36.285144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:36.285573 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:36.285555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48f4c58e-85c1-435a-bd62-30c0950be9ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:36.287345 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:36.287326 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48f4c58e-85c1-435a-bd62-30c0950be9ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-57t7d\" (UID: \"48f4c58e-85c1-435a-bd62-30c0950be9ba\") " pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:36.380647 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:36.380610 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-57t7d" Jul 21 20:18:36.524015 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:36.523963 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-vsghl" podStartSLOduration=13.523925166 podStartE2EDuration="13.523925166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:24.589074651 +0000 UTC m=+259.269201089" watchObservedRunningTime="2026-07-21 20:18:36.523925166 +0000 UTC m=+271.204051603" Jul 21 20:18:36.524565 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:36.524550 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-57t7d"] Jul 21 20:18:36.527107 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:36.527080 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f4c58e_85c1_435a_bd62_30c0950be9ba.slice/crio-545fac792682d0a47c717c620f0a67acc30716b4fbb2dcd51d2c2e6c0d85c9c5 WatchSource:0}: Error finding container 545fac792682d0a47c717c620f0a67acc30716b4fbb2dcd51d2c2e6c0d85c9c5: Status 404 returned error can't find the container with id 545fac792682d0a47c717c620f0a67acc30716b4fbb2dcd51d2c2e6c0d85c9c5 Jul 21 20:18:36.599577 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:36.599502 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57t7d" event={"ID":"48f4c58e-85c1-435a-bd62-30c0950be9ba","Type":"ContainerStarted","Data":"545fac792682d0a47c717c620f0a67acc30716b4fbb2dcd51d2c2e6c0d85c9c5"} Jul 21 20:18:41.619983 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:41.619933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57t7d" event={"ID":"48f4c58e-85c1-435a-bd62-30c0950be9ba","Type":"ContainerStarted","Data":"5f2c664523fb5511e6ad81d20cdc489ff79581cc5678f5a181c2e43d543910ab"} Jul 21 20:18:45.635095 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:45.635055 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57t7d" event={"ID":"48f4c58e-85c1-435a-bd62-30c0950be9ba","Type":"ContainerStarted","Data":"ec1956e4f200a400136562f9cb9a789fc48740481cda86bef7af88b82550148b"} Jul 21 20:18:46.294891 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:46.294849 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-g29mm" podUID="9bde1aba-deb0-4a77-898b-d3a48bcfca67" Jul 21 20:18:46.294891 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:18:46.294854 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" podUID="7fda0385-d801-453a-ab06-bcac6b323a07" Jul 21 20:18:46.638985 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:46.638952 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:18:46.639462 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:46.638958 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g29mm" Jul 21 20:18:47.077691 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:47.077592 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:47.077691 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:47.077679 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:47.080317 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:47.080291 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0af8ce4b-1621-4236-87e4-3705792ab31b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-r5j5b\" (UID: \"0af8ce4b-1621-4236-87e4-3705792ab31b\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:47.080453 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:47.080391 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-vddm7\" (UID: \"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:47.327985 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:47.327889 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" Jul 21 20:18:47.336108 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:47.336056 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" Jul 21 20:18:48.552318 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:48.552215 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b"] Jul 21 20:18:48.555389 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:48.555365 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af8ce4b_1621_4236_87e4_3705792ab31b.slice/crio-7cb60b795f4b507541ceea72269f9b901d5b48571e475a62cce5a0ff7f5679e8 WatchSource:0}: Error finding container 7cb60b795f4b507541ceea72269f9b901d5b48571e475a62cce5a0ff7f5679e8: Status 404 returned error can't find the container with id 7cb60b795f4b507541ceea72269f9b901d5b48571e475a62cce5a0ff7f5679e8 Jul 21 20:18:48.566518 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:48.566495 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7"] Jul 21 20:18:48.569459 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:48.569439 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc069c98e_f0ef_43cd_a1b1_bbda4e3a0eb8.slice/crio-abf3cf6cdea2e333b40b9da2c079b864f540b9aada8029ad590d0cd3dac0435b WatchSource:0}: Error finding container abf3cf6cdea2e333b40b9da2c079b864f540b9aada8029ad590d0cd3dac0435b: Status 404 returned error can't find the container with id abf3cf6cdea2e333b40b9da2c079b864f540b9aada8029ad590d0cd3dac0435b Jul 21 20:18:48.646758 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:48.646727 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" event={"ID":"0af8ce4b-1621-4236-87e4-3705792ab31b","Type":"ContainerStarted","Data":"7cb60b795f4b507541ceea72269f9b901d5b48571e475a62cce5a0ff7f5679e8"} Jul 21 20:18:48.647872 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:48.647844 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" event={"ID":"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8","Type":"ContainerStarted","Data":"abf3cf6cdea2e333b40b9da2c079b864f540b9aada8029ad590d0cd3dac0435b"} Jul 21 20:18:48.649666 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:48.649641 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-57t7d" event={"ID":"48f4c58e-85c1-435a-bd62-30c0950be9ba","Type":"ContainerStarted","Data":"644566f86526672e1b6f2d49c162f77691789b1a4c7b6daa77d74a9d2c372398"} Jul 21 20:18:48.669124 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:48.669081 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-57t7d" podStartSLOduration=16.831135888 podStartE2EDuration="28.669066602s" totalImagesPullingTime="11.837930714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 20:18:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:48.668016208 +0000 UTC m=+283.348142669" watchObservedRunningTime="2026-07-21 20:18:48.669066602 +0000 UTC m=+283.349193022" Jul 21 20:18:50.103254 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.103171 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:18:50.103254 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.103224 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:18:50.105833 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.105808 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9bde1aba-deb0-4a77-898b-d3a48bcfca67-metrics-tls\") pod \"dns-default-g29mm\" (UID: \"9bde1aba-deb0-4a77-898b-d3a48bcfca67\") " pod="openshift-dns/dns-default-g29mm" Jul 21 20:18:50.105988 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.105885 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"image-registry-664d77bf4-dsmcf\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:18:50.243407 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.243366 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k44nk\"" Jul 21 20:18:50.243407 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.243367 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v6nlp\"" Jul 21 20:18:50.250302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.250281 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g29mm" Jul 21 20:18:50.250449 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.250395 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:18:50.492612 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.492574 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g29mm"] Jul 21 20:18:50.495842 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:50.495813 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bde1aba_deb0_4a77_898b_d3a48bcfca67.slice/crio-73a61e1d76dd5f3658c19acccf1877e047119afb1dd06d24d0609892dcf22319 WatchSource:0}: Error finding container 73a61e1d76dd5f3658c19acccf1877e047119afb1dd06d24d0609892dcf22319: Status 404 returned error can't find the container with id 73a61e1d76dd5f3658c19acccf1877e047119afb1dd06d24d0609892dcf22319 Jul 21 20:18:50.517456 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.517424 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-664d77bf4-dsmcf"] Jul 21 20:18:50.520756 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:50.520733 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fda0385_d801_453a_ab06_bcac6b323a07.slice/crio-9dde252940fa3621f5969accdfc01539285d8436392adbdf6db096ab256d260f WatchSource:0}: Error finding container 9dde252940fa3621f5969accdfc01539285d8436392adbdf6db096ab256d260f: Status 404 returned error can't find the container with id 9dde252940fa3621f5969accdfc01539285d8436392adbdf6db096ab256d260f Jul 21 20:18:50.659336 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.659248 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" event={"ID":"0af8ce4b-1621-4236-87e4-3705792ab31b","Type":"ContainerStarted","Data":"97bc7e59d8bbbb41a88b01a1bdc5e6021a89ddb2925e8db7c9f31e4a02b343aa"} Jul 21 20:18:50.660803 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.660777 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" event={"ID":"c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8","Type":"ContainerStarted","Data":"a4d0f38007a25ab1156b56605ce1a6a252434c102ea5c9d08a768dc93f7a1566"} Jul 21 20:18:50.662242 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.662217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" event={"ID":"7fda0385-d801-453a-ab06-bcac6b323a07","Type":"ContainerStarted","Data":"fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae"} Jul 21 20:18:50.662373 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.662278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" event={"ID":"7fda0385-d801-453a-ab06-bcac6b323a07","Type":"ContainerStarted","Data":"9dde252940fa3621f5969accdfc01539285d8436392adbdf6db096ab256d260f"} Jul 21 20:18:50.662373 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.662305 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:18:50.663443 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.663418 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g29mm" event={"ID":"9bde1aba-deb0-4a77-898b-d3a48bcfca67","Type":"ContainerStarted","Data":"73a61e1d76dd5f3658c19acccf1877e047119afb1dd06d24d0609892dcf22319"} Jul 21 20:18:50.678907 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.678859 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-r5j5b" podStartSLOduration=33.819614022 podStartE2EDuration="35.678845755s" totalImagesPullingTime="1.859231733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:50.67767084 +0000 UTC m=+285.357797278" watchObservedRunningTime="2026-07-21 20:18:50.678845755 +0000 UTC m=+285.358972191" Jul 21 20:18:50.703453 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.703397 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" podStartSLOduration=284.703378145 podStartE2EDuration="4m44.703378145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:50.70186969 +0000 UTC m=+285.381996128" watchObservedRunningTime="2026-07-21 20:18:50.703378145 +0000 UTC m=+285.383504583" Jul 21 20:18:50.722144 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.722093 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-vddm7" podStartSLOduration=34.488085922 podStartE2EDuration="35.722078152s" totalImagesPullingTime="1.23399223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:50.720781232 +0000 UTC m=+285.400907715" watchObservedRunningTime="2026-07-21 20:18:50.722078152 +0000 UTC m=+285.402204588" Jul 21 20:18:50.948825 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.948750 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc"] Jul 21 20:18:50.953910 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.953883 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" Jul 21 20:18:50.956649 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.956202 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 20:18:50.956649 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.956490 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fl9jq\"" Jul 21 20:18:50.962135 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:50.962108 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc"] Jul 21 20:18:51.111233 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:51.111201 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ca5de520-0d52-4211-843e-ab7dc0ec2ea0-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-l7jzc\" (UID: \"ca5de520-0d52-4211-843e-ab7dc0ec2ea0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" Jul 21 20:18:51.211736 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:51.211645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ca5de520-0d52-4211-843e-ab7dc0ec2ea0-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-l7jzc\" (UID: \"ca5de520-0d52-4211-843e-ab7dc0ec2ea0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" Jul 21 20:18:51.214124 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:51.214099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ca5de520-0d52-4211-843e-ab7dc0ec2ea0-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-l7jzc\" (UID: \"ca5de520-0d52-4211-843e-ab7dc0ec2ea0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" Jul 21 20:18:51.266086 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:51.266051 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" Jul 21 20:18:51.409795 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:51.409750 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc"] Jul 21 20:18:51.412797 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:51.412761 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5de520_0d52_4211_843e_ab7dc0ec2ea0.slice/crio-9212a0933cc1cc01a254281745f8871f2a8e3323a3df7b6a6354d36235d725ef WatchSource:0}: Error finding container 9212a0933cc1cc01a254281745f8871f2a8e3323a3df7b6a6354d36235d725ef: Status 404 returned error can't find the container with id 9212a0933cc1cc01a254281745f8871f2a8e3323a3df7b6a6354d36235d725ef Jul 21 20:18:51.669565 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:51.669532 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" event={"ID":"ca5de520-0d52-4211-843e-ab7dc0ec2ea0","Type":"ContainerStarted","Data":"9212a0933cc1cc01a254281745f8871f2a8e3323a3df7b6a6354d36235d725ef"} Jul 21 20:18:52.623075 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.623043 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bc45556cb-q9sn8"] Jul 21 20:18:52.641685 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.641663 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bc45556cb-q9sn8"] Jul 21 20:18:52.641821 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.641756 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.644257 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.644238 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 20:18:52.644405 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.644267 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 20:18:52.644405 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.644304 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 20:18:52.644405 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.644333 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 20:18:52.644405 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.644397 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 20:18:52.645246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.645230 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-knxkg\"" Jul 21 20:18:52.645344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.645235 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 20:18:52.645344 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.645324 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 20:18:52.726012 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.725978 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-oauth-config\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.726189 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.726026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-serving-cert\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.726189 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.726055 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-config\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.726189 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.726140 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-oauth-serving-cert\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.726189 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.726161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdn99\" (UniqueName: \"kubernetes.io/projected/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-kube-api-access-vdn99\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.726189 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.726181 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-service-ca\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.827082 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.827048 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-oauth-serving-cert\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.827240 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.827096 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdn99\" (UniqueName: \"kubernetes.io/projected/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-kube-api-access-vdn99\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.827240 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.827124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-service-ca\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.827240 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.827173 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-oauth-config\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.827240 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.827209 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-serving-cert\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.827466 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.827242 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-config\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.827965 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.827886 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-oauth-serving-cert\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.828101 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.827991 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-service-ca\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.828220 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.828198 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-config\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.829986 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.829963 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-oauth-config\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.830251 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.830230 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-serving-cert\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.836450 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.836423 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdn99\" (UniqueName: \"kubernetes.io/projected/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-kube-api-access-vdn99\") pod \"console-bc45556cb-q9sn8\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:52.951807 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:52.951734 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:18:53.457059 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.457036 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bc45556cb-q9sn8"] Jul 21 20:18:53.460611 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:53.460583 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e56abe8_3aa8_4bfd_87f9_faf5ff159d47.slice/crio-e97087107dd31c057d493e23569567604394cc13bb32c210c277a896035b3c86 WatchSource:0}: Error finding container e97087107dd31c057d493e23569567604394cc13bb32c210c277a896035b3c86: Status 404 returned error can't find the container with id e97087107dd31c057d493e23569567604394cc13bb32c210c277a896035b3c86 Jul 21 20:18:53.677666 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.677629 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g29mm" event={"ID":"9bde1aba-deb0-4a77-898b-d3a48bcfca67","Type":"ContainerStarted","Data":"f5ee3877a03b2f1ed8832578b454357ef8d8416e2b6e37f936ecb38477bc0322"} Jul 21 20:18:53.678152 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.677692 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g29mm" event={"ID":"9bde1aba-deb0-4a77-898b-d3a48bcfca67","Type":"ContainerStarted","Data":"327fd3c2cb3792c0ebc6fc8e277f393645909173522bca47e76c4163dc9663da"} Jul 21 20:18:53.678152 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.677735 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g29mm" Jul 21 20:18:53.679085 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.679061 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc45556cb-q9sn8" event={"ID":"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47","Type":"ContainerStarted","Data":"e97087107dd31c057d493e23569567604394cc13bb32c210c277a896035b3c86"} Jul 21 20:18:53.680334 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.680313 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" event={"ID":"ca5de520-0d52-4211-843e-ab7dc0ec2ea0","Type":"ContainerStarted","Data":"fe17b3682de3f15420fd05a468f275bac56458c5b791988601f05f4215e9c969"} Jul 21 20:18:53.680516 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.680500 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" Jul 21 20:18:53.685370 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.685352 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" Jul 21 20:18:53.695552 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.695479 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g29mm" podStartSLOduration=250.866154863 podStartE2EDuration="4m13.695466128s" totalImagesPullingTime="2.829311265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:53.695002181 +0000 UTC m=+288.375128639" watchObservedRunningTime="2026-07-21 20:18:53.695466128 +0000 UTC m=+288.375592564" Jul 21 20:18:53.709310 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:53.709258 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l7jzc" podStartSLOduration=1.7954370929999999 podStartE2EDuration="3.709240798s" totalImagesPullingTime="1.913803705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:53.708545542 +0000 UTC m=+288.388671981" watchObservedRunningTime="2026-07-21 20:18:53.709240798 +0000 UTC m=+288.389367245" Jul 21 20:18:54.012514 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.012479 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl"] Jul 21 20:18:54.034712 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.034681 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl"] Jul 21 20:18:54.034845 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.034779 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.037562 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.037531 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 20:18:54.037723 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.037662 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nf746\"" Jul 21 20:18:54.037794 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.037739 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 20:18:54.037794 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.037739 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 20:18:54.136052 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.136000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nbx\" (UniqueName: \"kubernetes.io/projected/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-kube-api-access-q5nbx\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.136052 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.136054 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.136302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.136087 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.136302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.136229 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.236836 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.236801 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.237044 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.236855 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nbx\" (UniqueName: \"kubernetes.io/projected/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-kube-api-access-q5nbx\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.237044 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.237001 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.237170 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.237050 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.237519 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.237487 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.239374 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.239353 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.239477 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.239362 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.245343 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.245322 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nbx\" (UniqueName: \"kubernetes.io/projected/0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4-kube-api-access-q5nbx\") pod \"prometheus-operator-5945b5cfd-wl8dl\" (UID: \"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.344649 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.344576 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" Jul 21 20:18:54.517769 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.517734 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl"] Jul 21 20:18:54.520521 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:18:54.520492 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f22e398_ffef_42d5_8aeb_00a5cf4d5bd4.slice/crio-4e60aa19cfae6a7f46a839227aa464e8aeaf316f6f1393cc346adae2f0ecfe53 WatchSource:0}: Error finding container 4e60aa19cfae6a7f46a839227aa464e8aeaf316f6f1393cc346adae2f0ecfe53: Status 404 returned error can't find the container with id 4e60aa19cfae6a7f46a839227aa464e8aeaf316f6f1393cc346adae2f0ecfe53 Jul 21 20:18:54.687997 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:54.687957 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" event={"ID":"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4","Type":"ContainerStarted","Data":"4e60aa19cfae6a7f46a839227aa464e8aeaf316f6f1393cc346adae2f0ecfe53"} Jul 21 20:18:57.703105 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:57.703062 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc45556cb-q9sn8" event={"ID":"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47","Type":"ContainerStarted","Data":"c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3"} Jul 21 20:18:57.704848 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:57.704822 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" event={"ID":"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4","Type":"ContainerStarted","Data":"92ed3a6eaef360de7905b17a64c4ddaa8cb2acebef39feed5ee4e4f1793427f0"} Jul 21 20:18:57.730200 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:57.730138 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bc45556cb-q9sn8" podStartSLOduration=1.633448031 podStartE2EDuration="5.730116569s" totalImagesPullingTime="4.096668538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:57.730071897 +0000 UTC m=+292.410198335" watchObservedRunningTime="2026-07-21 20:18:57.730116569 +0000 UTC m=+292.410243010" Jul 21 20:18:58.709999 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:58.709957 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" event={"ID":"0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4","Type":"ContainerStarted","Data":"24fa63fe54b68097832fc6bc6250476e738eac7ebfef0c5833887100cdef09ba"} Jul 21 20:18:58.728238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:18:58.728194 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-wl8dl" podStartSLOduration=2.6875988790000003 podStartE2EDuration="5.728178783s" totalImagesPullingTime="3.040579904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:18:58.727446986 +0000 UTC m=+293.407573422" watchObservedRunningTime="2026-07-21 20:18:58.728178783 +0000 UTC m=+293.408305266" Jul 21 20:19:00.377875 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.377839 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4"] Jul 21 20:19:00.380321 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.380305 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.383382 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.383225 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 21 20:19:00.383382 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.383242 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 21 20:19:00.383578 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.383555 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-86mbc\"" Jul 21 20:19:00.394791 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.394765 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4"] Jul 21 20:19:00.396426 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.396405 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-ccqsj"] Jul 21 20:19:00.399346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.399329 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.401877 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.401858 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zpxqk\"" Jul 21 20:19:00.402156 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.402141 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 21 20:19:00.402406 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.402375 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 21 20:19:00.402590 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.402223 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 21 20:19:00.410057 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.410038 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-ccqsj"] Jul 21 20:19:00.419529 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.419508 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pql9x"] Jul 21 20:19:00.422057 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.422043 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.424107 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.424092 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xnqx8\"" Jul 21 20:19:00.424498 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.424480 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 20:19:00.424650 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.424629 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 20:19:00.424723 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.424694 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 20:19:00.494761 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.494736 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6921272b-a355-44c3-a0c7-73b972bbee34-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.494927 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.494785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6921272b-a355-44c3-a0c7-73b972bbee34-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.494927 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.494814 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg8bz\" (UniqueName: \"kubernetes.io/projected/6921272b-a355-44c3-a0c7-73b972bbee34-kube-api-access-lg8bz\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.494927 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.494910 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6921272b-a355-44c3-a0c7-73b972bbee34-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.596172 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-tls\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.596331 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596175 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.596331 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596244 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76ps\" (UniqueName: \"kubernetes.io/projected/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-api-access-j76ps\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.596331 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596306 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8349de3d-1a4c-42fe-bdba-ee4d76730314-metrics-client-ca\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.596445 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8349de3d-1a4c-42fe-bdba-ee4d76730314-sys\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.596445 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596363 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6921272b-a355-44c3-a0c7-73b972bbee34-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.596445 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596402 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.596445 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596429 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8349de3d-1a4c-42fe-bdba-ee4d76730314-root\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.596615 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596458 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg8bz\" (UniqueName: \"kubernetes.io/projected/6921272b-a355-44c3-a0c7-73b972bbee34-kube-api-access-lg8bz\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.596615 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596515 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a8b1b66-cf08-4305-a05e-82a17c308318-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.596615 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596602 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6921272b-a355-44c3-a0c7-73b972bbee34-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.596765 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596645 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.596765 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596680 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.596765 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596719 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6921272b-a355-44c3-a0c7-73b972bbee34-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.596765 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-wtmp\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.596976 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a8b1b66-cf08-4305-a05e-82a17c308318-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.596976 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596808 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-accelerators-collector-config\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.596976 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596885 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-textfile\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.596976 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.596913 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxzv\" (UniqueName: \"kubernetes.io/projected/8349de3d-1a4c-42fe-bdba-ee4d76730314-kube-api-access-tqxzv\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.597159 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.597090 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6921272b-a355-44c3-a0c7-73b972bbee34-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.598993 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.598959 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6921272b-a355-44c3-a0c7-73b972bbee34-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.599207 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.599187 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6921272b-a355-44c3-a0c7-73b972bbee34-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.604657 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.604597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg8bz\" (UniqueName: \"kubernetes.io/projected/6921272b-a355-44c3-a0c7-73b972bbee34-kube-api-access-lg8bz\") pod \"openshift-state-metrics-79f6958cdf-6p4g4\" (UID: \"6921272b-a355-44c3-a0c7-73b972bbee34\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.690550 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.690473 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" Jul 21 20:19:00.698309 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698282 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-textfile\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.698428 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698321 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxzv\" (UniqueName: \"kubernetes.io/projected/8349de3d-1a4c-42fe-bdba-ee4d76730314-kube-api-access-tqxzv\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.698428 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698347 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-tls\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.698428 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698372 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.698428 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76ps\" (UniqueName: \"kubernetes.io/projected/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-api-access-j76ps\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.698637 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8349de3d-1a4c-42fe-bdba-ee4d76730314-metrics-client-ca\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.698637 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8349de3d-1a4c-42fe-bdba-ee4d76730314-sys\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.698637 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698482 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.698637 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698508 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8349de3d-1a4c-42fe-bdba-ee4d76730314-root\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.698637 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a8b1b66-cf08-4305-a05e-82a17c308318-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.698637 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.698637 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698621 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.699041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-wtmp\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.699041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698680 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a8b1b66-cf08-4305-a05e-82a17c308318-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.699041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.698709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-accelerators-collector-config\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.699319 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.699298 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-accelerators-collector-config\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.700140 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.699524 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-textfile\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.700140 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.700045 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8349de3d-1a4c-42fe-bdba-ee4d76730314-root\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.700297 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:19:00.700145 2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 21 20:19:00.700297 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:19:00.700198 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-tls podName:1a8b1b66-cf08-4305-a05e-82a17c308318 nodeName:}" failed. No retries permitted until 2026-07-21 20:19:01.200179836 +0000 UTC m=+295.880306259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-ccqsj" (UID: "1a8b1b66-cf08-4305-a05e-82a17c308318") : secret "kube-state-metrics-tls" not found Jul 21 20:19:00.700472 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.700457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8349de3d-1a4c-42fe-bdba-ee4d76730314-sys\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.701021 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.701003 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8349de3d-1a4c-42fe-bdba-ee4d76730314-metrics-client-ca\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.701759 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.701342 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.701759 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.701648 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a8b1b66-cf08-4305-a05e-82a17c308318-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.701924 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.701787 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-wtmp\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.702080 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.702054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1a8b1b66-cf08-4305-a05e-82a17c308318-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.704346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.704319 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.704434 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.704323 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-tls\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.706693 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.706674 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8349de3d-1a4c-42fe-bdba-ee4d76730314-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.710635 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.710615 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76ps\" (UniqueName: \"kubernetes.io/projected/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-api-access-j76ps\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:00.711341 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.711322 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxzv\" (UniqueName: \"kubernetes.io/projected/8349de3d-1a4c-42fe-bdba-ee4d76730314-kube-api-access-tqxzv\") pod \"node-exporter-pql9x\" (UID: \"8349de3d-1a4c-42fe-bdba-ee4d76730314\") " pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.730524 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.730500 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pql9x" Jul 21 20:19:00.743298 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:19:00.743272 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8349de3d_1a4c_42fe_bdba_ee4d76730314.slice/crio-531a390ebb891203cc753221f1e00a2e83501ae265e0687bc00f215d0f1dec6c WatchSource:0}: Error finding container 531a390ebb891203cc753221f1e00a2e83501ae265e0687bc00f215d0f1dec6c: Status 404 returned error can't find the container with id 531a390ebb891203cc753221f1e00a2e83501ae265e0687bc00f215d0f1dec6c Jul 21 20:19:00.891409 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.891379 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86cd5fbb68-s2d8r"] Jul 21 20:19:00.892206 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:19:00.892176 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6921272b_a355_44c3_a0c7_73b972bbee34.slice/crio-24bbfcf113f4c9792f32300f1632cdc98f17ee030af8b8ccdec61b175e5a49bc WatchSource:0}: Error finding container 24bbfcf113f4c9792f32300f1632cdc98f17ee030af8b8ccdec61b175e5a49bc: Status 404 returned error can't find the container with id 24bbfcf113f4c9792f32300f1632cdc98f17ee030af8b8ccdec61b175e5a49bc Jul 21 20:19:00.895451 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.895434 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4"] Jul 21 20:19:00.895527 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.895518 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:00.906164 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.906142 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 20:19:00.934423 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:00.934401 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cd5fbb68-s2d8r"] Jul 21 20:19:01.000605 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.000573 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-trusted-ca-bundle\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.000753 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.000646 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-config\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.000753 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.000723 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-oauth-config\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.000836 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.000776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-serving-cert\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.000836 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.000803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-oauth-serving-cert\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.000836 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.000826 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxkn5\" (UniqueName: \"kubernetes.io/projected/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-kube-api-access-dxkn5\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.000994 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.000878 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-service-ca\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.102514 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.102453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-config\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.102989 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.102619 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-oauth-config\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.102989 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.102677 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-serving-cert\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.102989 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.102711 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-oauth-serving-cert\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.102989 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.102741 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxkn5\" (UniqueName: \"kubernetes.io/projected/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-kube-api-access-dxkn5\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.102989 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.102785 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-service-ca\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.102989 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.102852 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-trusted-ca-bundle\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.107407 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.107381 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-trusted-ca-bundle\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.108227 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.108200 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-oauth-serving-cert\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.108345 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.108264 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-service-ca\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.108450 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.108422 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-config\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.109403 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.109380 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-serving-cert\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.109618 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.109602 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-oauth-config\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.116454 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.116431 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxkn5\" (UniqueName: \"kubernetes.io/projected/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-kube-api-access-dxkn5\") pod \"console-86cd5fbb68-s2d8r\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.203403 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.203326 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:01.205521 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.205502 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1a8b1b66-cf08-4305-a05e-82a17c308318-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-ccqsj\" (UID: \"1a8b1b66-cf08-4305-a05e-82a17c308318\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:01.208450 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.208424 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:01.308410 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.308384 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" Jul 21 20:19:01.342742 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.342713 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86cd5fbb68-s2d8r"] Jul 21 20:19:01.346885 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:19:01.346855 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c1fbd2_2af2_40d4_8d20_be78ba9bf89b.slice/crio-812102f4926d5e0e351e0d4988a8c2a54db1c584d5b0bfabb4f2bf864fce547b WatchSource:0}: Error finding container 812102f4926d5e0e351e0d4988a8c2a54db1c584d5b0bfabb4f2bf864fce547b: Status 404 returned error can't find the container with id 812102f4926d5e0e351e0d4988a8c2a54db1c584d5b0bfabb4f2bf864fce547b Jul 21 20:19:01.490808 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.490678 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-ccqsj"] Jul 21 20:19:01.502131 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:19:01.502093 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8b1b66_cf08_4305_a05e_82a17c308318.slice/crio-bdf44c6b609c86cdeed02c4652e20caeb970a48024514fe4ff517076f545938d WatchSource:0}: Error finding container bdf44c6b609c86cdeed02c4652e20caeb970a48024514fe4ff517076f545938d: Status 404 returned error can't find the container with id bdf44c6b609c86cdeed02c4652e20caeb970a48024514fe4ff517076f545938d Jul 21 20:19:01.724960 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.724897 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pql9x" event={"ID":"8349de3d-1a4c-42fe-bdba-ee4d76730314","Type":"ContainerStarted","Data":"531a390ebb891203cc753221f1e00a2e83501ae265e0687bc00f215d0f1dec6c"} Jul 21 20:19:01.727779 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.727642 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" event={"ID":"6921272b-a355-44c3-a0c7-73b972bbee34","Type":"ContainerStarted","Data":"c4e4b83919b77033b7ae665f8980f5782638350f662ccf9fe8a68fd74d5f1e25"} Jul 21 20:19:01.727779 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.727708 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" event={"ID":"6921272b-a355-44c3-a0c7-73b972bbee34","Type":"ContainerStarted","Data":"6211ee372ce999a551114bddd98c26c5682ee1cceb53a60447671f05b3486b9a"} Jul 21 20:19:01.727779 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.727741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" event={"ID":"6921272b-a355-44c3-a0c7-73b972bbee34","Type":"ContainerStarted","Data":"24bbfcf113f4c9792f32300f1632cdc98f17ee030af8b8ccdec61b175e5a49bc"} Jul 21 20:19:01.729863 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.729838 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cd5fbb68-s2d8r" event={"ID":"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b","Type":"ContainerStarted","Data":"4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a"} Jul 21 20:19:01.730018 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.729883 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cd5fbb68-s2d8r" event={"ID":"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b","Type":"ContainerStarted","Data":"812102f4926d5e0e351e0d4988a8c2a54db1c584d5b0bfabb4f2bf864fce547b"} Jul 21 20:19:01.731663 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.731605 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" event={"ID":"1a8b1b66-cf08-4305-a05e-82a17c308318","Type":"ContainerStarted","Data":"bdf44c6b609c86cdeed02c4652e20caeb970a48024514fe4ff517076f545938d"} Jul 21 20:19:01.752182 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:01.752034 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86cd5fbb68-s2d8r" podStartSLOduration=1.75201391 podStartE2EDuration="1.75201391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:19:01.749512439 +0000 UTC m=+296.429638879" watchObservedRunningTime="2026-07-21 20:19:01.75201391 +0000 UTC m=+296.432140353" Jul 21 20:19:01.800304 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:19:01.800258 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-f57lq" podUID="011ea63a-4a29-4f3e-9710-4b9b299ba69c" Jul 21 20:19:02.737168 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:02.737134 2567 generic.go:356] "Generic (PLEG): container finished" podID="8349de3d-1a4c-42fe-bdba-ee4d76730314" containerID="18ee8247349d3d5aea645f526433ebbf6ac72de900b189d1498472f12450d40a" exitCode=0 Jul 21 20:19:02.737640 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:02.737227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pql9x" event={"ID":"8349de3d-1a4c-42fe-bdba-ee4d76730314","Type":"ContainerDied","Data":"18ee8247349d3d5aea645f526433ebbf6ac72de900b189d1498472f12450d40a"} Jul 21 20:19:02.739382 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:02.739360 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" event={"ID":"6921272b-a355-44c3-a0c7-73b972bbee34","Type":"ContainerStarted","Data":"1abaaf9e329de00a8405f03b118c156562697ccaea36f859c286ad87e60e2d37"} Jul 21 20:19:02.838835 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:02.838794 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-6p4g4" podStartSLOduration=1.573554269 podStartE2EDuration="2.838778106s" totalImagesPullingTime="1.265223837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:19:02.83809139 +0000 UTC m=+297.518217849" watchObservedRunningTime="2026-07-21 20:19:02.838778106 +0000 UTC m=+297.518904543" Jul 21 20:19:02.952150 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:02.952112 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:19:02.952324 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:02.952181 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:19:02.953662 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:02.953618 2567 patch_prober.go:28] interesting pod/console-bc45556cb-q9sn8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Jul 21 20:19:02.953825 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:02.953684 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-bc45556cb-q9sn8" podUID="1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" containerName="console" probeResult="failure" output="Get \"https://10.132.0.20:8443/health\": dial tcp 10.132.0.20:8443: connect: connection refused" Jul 21 20:19:03.690752 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:03.690722 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g29mm" Jul 21 20:19:03.746023 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:03.745988 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" event={"ID":"1a8b1b66-cf08-4305-a05e-82a17c308318","Type":"ContainerStarted","Data":"51fea16638fcfaaf5407ccc829dd93e76112d76a5f20cc6d083cac9d404e0c75"} Jul 21 20:19:03.746455 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:03.746050 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" event={"ID":"1a8b1b66-cf08-4305-a05e-82a17c308318","Type":"ContainerStarted","Data":"5b2fd8512cf5eb6418bd43626b361e642c7e7c9a228a6d50f2acf46d58aba06e"} Jul 21 20:19:03.746455 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:03.746085 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" event={"ID":"1a8b1b66-cf08-4305-a05e-82a17c308318","Type":"ContainerStarted","Data":"116239e85940f0585c3fe6eeccff544d985874e9c60ec7f64b2697e259566ac5"} Jul 21 20:19:03.748775 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:03.748748 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pql9x" event={"ID":"8349de3d-1a4c-42fe-bdba-ee4d76730314","Type":"ContainerStarted","Data":"836eea8c09f71dbbd424c4e714160256735d79a82abc1efe45f2ac94a7aa1c15"} Jul 21 20:19:03.748904 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:03.748802 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pql9x" event={"ID":"8349de3d-1a4c-42fe-bdba-ee4d76730314","Type":"ContainerStarted","Data":"f08feae58405ab39aad747746606235324abb4835e0ec7a7af51864fed64cb4e"} Jul 21 20:19:03.768341 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:03.768289 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-ccqsj" podStartSLOduration=2.148603306 podStartE2EDuration="3.768274054s" totalImagesPullingTime="1.619670748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:19:03.767049984 +0000 UTC m=+298.447176425" watchObservedRunningTime="2026-07-21 20:19:03.768274054 +0000 UTC m=+298.448400491" Jul 21 20:19:03.792383 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:03.792333 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pql9x" podStartSLOduration=2.527362541 podStartE2EDuration="3.792315881s" totalImagesPullingTime="1.194561215s" totalInitContainerRuntime="70.392125ms" isStatefulPod=true podCreationTimestamp="2026-07-21 20:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:19:03.790709318 +0000 UTC m=+298.470835775" watchObservedRunningTime="2026-07-21 20:19:03.792315881 +0000 UTC m=+298.472442351" Jul 21 20:19:04.774714 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.774679 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p"] Jul 21 20:19:04.778343 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.778321 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.781858 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.781837 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 20:19:04.781858 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.781857 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6sg7o45nqiic4\"" Jul 21 20:19:04.782091 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.781883 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 21 20:19:04.782091 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.781905 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8xlzb\"" Jul 21 20:19:04.782091 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.781971 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 21 20:19:04.782091 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.782036 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 21 20:19:04.788347 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.788323 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p"] Jul 21 20:19:04.834171 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.834135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/03255225-ed9a-4f13-a6e9-657af5a34306-secret-metrics-server-tls\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.834381 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.834209 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/03255225-ed9a-4f13-a6e9-657af5a34306-metrics-server-audit-profiles\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.834381 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.834279 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/03255225-ed9a-4f13-a6e9-657af5a34306-client-ca-bundle\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.834381 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.834310 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03255225-ed9a-4f13-a6e9-657af5a34306-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.834544 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.834501 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqvb9\" (UniqueName: \"kubernetes.io/projected/03255225-ed9a-4f13-a6e9-657af5a34306-kube-api-access-kqvb9\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.834608 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.834577 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/03255225-ed9a-4f13-a6e9-657af5a34306-secret-metrics-server-client-certs\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.834760 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.834738 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/03255225-ed9a-4f13-a6e9-657af5a34306-audit-log\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.935736 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.935694 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/03255225-ed9a-4f13-a6e9-657af5a34306-metrics-server-audit-profiles\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.935933 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.935753 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/03255225-ed9a-4f13-a6e9-657af5a34306-client-ca-bundle\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.935933 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.935789 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03255225-ed9a-4f13-a6e9-657af5a34306-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.935933 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.935859 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqvb9\" (UniqueName: \"kubernetes.io/projected/03255225-ed9a-4f13-a6e9-657af5a34306-kube-api-access-kqvb9\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.935933 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.935887 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/03255225-ed9a-4f13-a6e9-657af5a34306-secret-metrics-server-client-certs\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.935933 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.935930 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/03255225-ed9a-4f13-a6e9-657af5a34306-audit-log\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.936234 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.935989 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/03255225-ed9a-4f13-a6e9-657af5a34306-secret-metrics-server-tls\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.936377 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.936337 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/03255225-ed9a-4f13-a6e9-657af5a34306-audit-log\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.936596 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.936571 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03255225-ed9a-4f13-a6e9-657af5a34306-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.936794 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.936774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/03255225-ed9a-4f13-a6e9-657af5a34306-metrics-server-audit-profiles\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.938358 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.938330 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/03255225-ed9a-4f13-a6e9-657af5a34306-client-ca-bundle\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.938445 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.938376 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/03255225-ed9a-4f13-a6e9-657af5a34306-secret-metrics-server-tls\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.938489 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.938442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/03255225-ed9a-4f13-a6e9-657af5a34306-secret-metrics-server-client-certs\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:04.944438 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:04.944421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqvb9\" (UniqueName: \"kubernetes.io/projected/03255225-ed9a-4f13-a6e9-657af5a34306-kube-api-access-kqvb9\") pod \"metrics-server-54ff9d6fdf-gmm5p\" (UID: \"03255225-ed9a-4f13-a6e9-657af5a34306\") " pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:05.088471 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.088391 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:05.235784 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.235693 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p"] Jul 21 20:19:05.239441 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:19:05.239401 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03255225_ed9a_4f13_a6e9_657af5a34306.slice/crio-94125b27a759457f5b98ee9843560fd23bdd69ff67b4c6928674333ba174427c WatchSource:0}: Error finding container 94125b27a759457f5b98ee9843560fd23bdd69ff67b4c6928674333ba174427c: Status 404 returned error can't find the container with id 94125b27a759457f5b98ee9843560fd23bdd69ff67b4c6928674333ba174427c Jul 21 20:19:05.630341 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.630305 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff"] Jul 21 20:19:05.634898 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.634873 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.637534 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.637495 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 21 20:19:05.637534 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.637508 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 21 20:19:05.637701 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.637571 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 21 20:19:05.637701 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.637581 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 21 20:19:05.637701 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.637581 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lb4h2\"" Jul 21 20:19:05.637967 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.637933 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 21 20:19:05.644866 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.643892 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 21 20:19:05.651300 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.651276 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff"] Jul 21 20:19:05.683513 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.683489 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:19:05.683820 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.683804 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:19:05.687299 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.687279 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 20:19:05.745009 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.744970 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-telemeter-client-tls\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.745009 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.745012 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-federate-client-tls\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.745273 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.745058 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdb5\" (UniqueName: \"kubernetes.io/projected/9c44d643-91df-41d0-9a33-f27572987b54-kube-api-access-srdb5\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.745273 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.745087 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c44d643-91df-41d0-9a33-f27572987b54-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.745273 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.745155 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c44d643-91df-41d0-9a33-f27572987b54-serving-certs-ca-bundle\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.745273 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.745176 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c44d643-91df-41d0-9a33-f27572987b54-metrics-client-ca\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.745273 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.745241 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-secret-telemeter-client\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.745448 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.745273 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.757821 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.757785 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" event={"ID":"03255225-ed9a-4f13-a6e9-657af5a34306","Type":"ContainerStarted","Data":"94125b27a759457f5b98ee9843560fd23bdd69ff67b4c6928674333ba174427c"} Jul 21 20:19:05.846470 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.846315 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-secret-telemeter-client\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.846470 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.846359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.846470 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.846399 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-telemeter-client-tls\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.846470 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.846415 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-federate-client-tls\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.846470 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.846443 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srdb5\" (UniqueName: \"kubernetes.io/projected/9c44d643-91df-41d0-9a33-f27572987b54-kube-api-access-srdb5\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.846470 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.846463 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c44d643-91df-41d0-9a33-f27572987b54-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.846504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c44d643-91df-41d0-9a33-f27572987b54-serving-certs-ca-bundle\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.846531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c44d643-91df-41d0-9a33-f27572987b54-metrics-client-ca\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.847302 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c44d643-91df-41d0-9a33-f27572987b54-metrics-client-ca\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.847378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c44d643-91df-41d0-9a33-f27572987b54-serving-certs-ca-bundle\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.847465 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c44d643-91df-41d0-9a33-f27572987b54-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.849065 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-federate-client-tls\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.849123 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-telemeter-client-tls\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.849173 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.849174 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c44d643-91df-41d0-9a33-f27572987b54-secret-telemeter-client\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.863238 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.859956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdb5\" (UniqueName: \"kubernetes.io/projected/9c44d643-91df-41d0-9a33-f27572987b54-kube-api-access-srdb5\") pod \"telemeter-client-d4bd6fd4b-9wmff\" (UID: \"9c44d643-91df-41d0-9a33-f27572987b54\") " pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:05.947415 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:05.947306 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" Jul 21 20:19:06.111694 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:06.111662 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff"] Jul 21 20:19:06.115210 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:19:06.115183 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c44d643_91df_41d0_9a33_f27572987b54.slice/crio-c9b75694bc96879fb615c0d69cec57d79edec3ff5d3ed30bf969db8890931af8 WatchSource:0}: Error finding container c9b75694bc96879fb615c0d69cec57d79edec3ff5d3ed30bf969db8890931af8: Status 404 returned error can't find the container with id c9b75694bc96879fb615c0d69cec57d79edec3ff5d3ed30bf969db8890931af8 Jul 21 20:19:06.765249 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:06.765205 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" event={"ID":"9c44d643-91df-41d0-9a33-f27572987b54","Type":"ContainerStarted","Data":"c9b75694bc96879fb615c0d69cec57d79edec3ff5d3ed30bf969db8890931af8"} Jul 21 20:19:07.432800 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.432758 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bc45556cb-q9sn8"] Jul 21 20:19:07.465385 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.462543 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-784679754-lbq5j"] Jul 21 20:19:07.466687 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.466657 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.474859 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.474833 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784679754-lbq5j"] Jul 21 20:19:07.552235 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.552204 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-664d77bf4-dsmcf"] Jul 21 20:19:07.556799 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.556765 2567 patch_prober.go:28] interesting pod/image-registry-664d77bf4-dsmcf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 21 20:19:07.556937 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.556819 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" podUID="7fda0385-d801-453a-ab06-bcac6b323a07" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 20:19:07.568193 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.568165 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-service-ca\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.568351 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.568202 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8sd\" (UniqueName: \"kubernetes.io/projected/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-kube-api-access-lt8sd\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.568351 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.568307 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-serving-cert\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.568351 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.568338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-oauth-serving-cert\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.568512 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.568372 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-oauth-config\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.568512 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.568423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-config\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.568512 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.568447 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-trusted-ca-bundle\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.669348 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.669309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-config\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.669348 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.669345 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-trusted-ca-bundle\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.669577 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.669393 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-service-ca\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.669577 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.669409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8sd\" (UniqueName: \"kubernetes.io/projected/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-kube-api-access-lt8sd\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.669577 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.669484 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-serving-cert\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.669577 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.669515 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-oauth-serving-cert\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.669577 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.669556 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-oauth-config\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.670188 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.670156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-config\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.670393 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.670367 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-service-ca\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.670466 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.670367 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-oauth-serving-cert\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.670466 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.670450 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-trusted-ca-bundle\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.672405 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.672380 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-serving-cert\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.673426 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.673385 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-oauth-config\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.678530 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.678502 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8sd\" (UniqueName: \"kubernetes.io/projected/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-kube-api-access-lt8sd\") pod \"console-784679754-lbq5j\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.772677 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.772599 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" event={"ID":"03255225-ed9a-4f13-a6e9-657af5a34306","Type":"ContainerStarted","Data":"8545e09bc105968eab005c9ff1727f676c07a72b83271ac9ad63814b23115a0d"} Jul 21 20:19:07.775984 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.775963 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:07.793896 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.793852 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" podStartSLOduration=2.110665441 podStartE2EDuration="3.793835217s" totalImagesPullingTime="1.683169776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:19:07.792596549 +0000 UTC m=+302.472723019" watchObservedRunningTime="2026-07-21 20:19:07.793835217 +0000 UTC m=+302.473961654" Jul 21 20:19:07.913738 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:07.913692 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784679754-lbq5j"] Jul 21 20:19:07.916866 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:19:07.916837 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f61e22_e29b_404e_8e3c_e8d8c92c35cb.slice/crio-e915ce6930c5bfbeb34f923109ffa8612192a182c94a0f92aa744af5fa247124 WatchSource:0}: Error finding container e915ce6930c5bfbeb34f923109ffa8612192a182c94a0f92aa744af5fa247124: Status 404 returned error can't find the container with id e915ce6930c5bfbeb34f923109ffa8612192a182c94a0f92aa744af5fa247124 Jul 21 20:19:08.779515 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:08.779481 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784679754-lbq5j" event={"ID":"66f61e22-e29b-404e-8e3c-e8d8c92c35cb","Type":"ContainerStarted","Data":"9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b"} Jul 21 20:19:08.780003 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:08.779539 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784679754-lbq5j" event={"ID":"66f61e22-e29b-404e-8e3c-e8d8c92c35cb","Type":"ContainerStarted","Data":"e915ce6930c5bfbeb34f923109ffa8612192a182c94a0f92aa744af5fa247124"} Jul 21 20:19:08.797313 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:08.797266 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-784679754-lbq5j" podStartSLOduration=1.797249432 podStartE2EDuration="1.797249432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:19:08.796207811 +0000 UTC m=+303.476334242" watchObservedRunningTime="2026-07-21 20:19:08.797249432 +0000 UTC m=+303.477375847" Jul 21 20:19:09.027729 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:09.027712 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 20:19:09.786165 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:09.786121 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" event={"ID":"9c44d643-91df-41d0-9a33-f27572987b54","Type":"ContainerStarted","Data":"fa975d0385c4bce3787ed9d3fde68dc07712dda2e145fbd3a1e71220e1b98033"} Jul 21 20:19:10.792165 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:10.792129 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" event={"ID":"9c44d643-91df-41d0-9a33-f27572987b54","Type":"ContainerStarted","Data":"5921f480d2235d88d42315d14881a9b13671a9eccdb3807e02cd01882f6ebb56"} Jul 21 20:19:10.792630 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:10.792192 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" event={"ID":"9c44d643-91df-41d0-9a33-f27572987b54","Type":"ContainerStarted","Data":"e852066d2e1d669f065328a1f526a09aef204572755aa9e3ca14f3668b08ad7f"} Jul 21 20:19:10.815840 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:10.815785 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d4bd6fd4b-9wmff" podStartSLOduration=2.035172633 podStartE2EDuration="5.815766514s" totalImagesPullingTime="3.780593881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:19:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:19:10.814329937 +0000 UTC m=+305.494456389" watchObservedRunningTime="2026-07-21 20:19:10.815766514 +0000 UTC m=+305.495892951" Jul 21 20:19:11.208735 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.208675 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:11.208900 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.208754 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:11.213438 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.213413 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:11.594500 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.594412 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cd5fbb68-s2d8r"] Jul 21 20:19:11.625999 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.625959 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66b6c546f8-bkz24"] Jul 21 20:19:11.629411 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.629393 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.641818 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.641795 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b6c546f8-bkz24"] Jul 21 20:19:11.709785 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.709750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtn4t\" (UniqueName: \"kubernetes.io/projected/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-kube-api-access-mtn4t\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.709986 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.709796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-oauth-config\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.709986 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.709836 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-oauth-serving-cert\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.709986 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.709862 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-trusted-ca-bundle\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.709986 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.709882 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-config\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.709986 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.709910 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-service-ca\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.709986 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.709965 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-serving-cert\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.806746 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.806714 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:11.810787 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.810762 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-oauth-serving-cert\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.810787 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.810793 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-trusted-ca-bundle\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.810993 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.810811 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-config\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.810993 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.810934 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-service-ca\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.811072 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.811016 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-serving-cert\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.811119 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.811076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtn4t\" (UniqueName: \"kubernetes.io/projected/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-kube-api-access-mtn4t\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.811119 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.811111 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-oauth-config\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.811750 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.811722 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-oauth-serving-cert\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.811750 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.811744 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-service-ca\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.811924 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.811754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-trusted-ca-bundle\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.812152 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.812118 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-config\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.813582 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.813561 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-serving-cert\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.813685 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.813598 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-oauth-config\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.822270 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.822246 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtn4t\" (UniqueName: \"kubernetes.io/projected/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-kube-api-access-mtn4t\") pod \"console-66b6c546f8-bkz24\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:11.939913 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:11.939869 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:12.080544 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:12.080515 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b6c546f8-bkz24"] Jul 21 20:19:12.082916 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:19:12.082892 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289eed1f_113e_4061_b4ba_1f8fb0e1ba40.slice/crio-0cf7d21a7a9d7928e6c389cbff77a5ef47b492b14abea0815d6161aa47bce18a WatchSource:0}: Error finding container 0cf7d21a7a9d7928e6c389cbff77a5ef47b492b14abea0815d6161aa47bce18a: Status 404 returned error can't find the container with id 0cf7d21a7a9d7928e6c389cbff77a5ef47b492b14abea0815d6161aa47bce18a Jul 21 20:19:12.798745 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:12.798714 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:19:12.805976 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:12.805739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b6c546f8-bkz24" event={"ID":"289eed1f-113e-4061-b4ba-1f8fb0e1ba40","Type":"ContainerStarted","Data":"d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d"} Jul 21 20:19:12.805976 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:12.805831 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b6c546f8-bkz24" event={"ID":"289eed1f-113e-4061-b4ba-1f8fb0e1ba40","Type":"ContainerStarted","Data":"0cf7d21a7a9d7928e6c389cbff77a5ef47b492b14abea0815d6161aa47bce18a"} Jul 21 20:19:12.843206 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:12.843154 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66b6c546f8-bkz24" podStartSLOduration=1.843137346 podStartE2EDuration="1.843137346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:19:12.842234542 +0000 UTC m=+307.522360984" watchObservedRunningTime="2026-07-21 20:19:12.843137346 +0000 UTC m=+307.523263783" Jul 21 20:19:17.557417 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:17.557382 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:19:17.776380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:17.776336 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:17.776568 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:17.776404 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:17.782273 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:17.782248 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:17.830502 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:17.830418 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:21.940256 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:21.940216 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:21.940710 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:21.940276 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:21.945203 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:21.945181 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:22.848847 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:22.848822 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:19:22.901598 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:22.901566 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-784679754-lbq5j"] Jul 21 20:19:25.088919 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:25.088882 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:25.089507 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:25.089001 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:32.453187 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.453146 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bc45556cb-q9sn8" podUID="1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" containerName="console" containerID="cri-o://c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3" gracePeriod=15 Jul 21 20:19:32.575524 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.575490 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" podUID="7fda0385-d801-453a-ab06-bcac6b323a07" containerName="registry" containerID="cri-o://fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae" gracePeriod=30 Jul 21 20:19:32.709986 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.709919 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:19:32.808445 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.808418 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdn99\" (UniqueName: \"kubernetes.io/projected/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-kube-api-access-vdn99\") pod \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " Jul 21 20:19:32.808640 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.808506 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-oauth-serving-cert\") pod \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " Jul 21 20:19:32.808640 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.808539 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-serving-cert\") pod \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " Jul 21 20:19:32.808640 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.808580 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-service-ca\") pod \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " Jul 21 20:19:32.808640 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.808631 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-config\") pod \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " Jul 21 20:19:32.808841 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.808678 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-oauth-config\") pod \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\" (UID: \"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47\") " Jul 21 20:19:32.809011 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.808958 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" (UID: "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:32.809143 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.809025 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-service-ca" (OuterVolumeSpecName: "service-ca") pod "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" (UID: "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:32.809143 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.809060 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-config" (OuterVolumeSpecName: "console-config") pod "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" (UID: "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:32.810881 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.810850 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-kube-api-access-vdn99" (OuterVolumeSpecName: "kube-api-access-vdn99") pod "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" (UID: "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47"). InnerVolumeSpecName "kube-api-access-vdn99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:19:32.811006 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.810883 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" (UID: "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:19:32.811006 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.810988 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" (UID: "1e56abe8-3aa8-4bfd-87f9-faf5ff159d47"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:19:32.828128 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.828109 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:19:32.892270 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.892235 2567 generic.go:356] "Generic (PLEG): container finished" podID="7fda0385-d801-453a-ab06-bcac6b323a07" containerID="fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae" exitCode=0 Jul 21 20:19:32.892429 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.892280 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" Jul 21 20:19:32.892429 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.892314 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" event={"ID":"7fda0385-d801-453a-ab06-bcac6b323a07","Type":"ContainerDied","Data":"fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae"} Jul 21 20:19:32.892429 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.892356 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-664d77bf4-dsmcf" event={"ID":"7fda0385-d801-453a-ab06-bcac6b323a07","Type":"ContainerDied","Data":"9dde252940fa3621f5969accdfc01539285d8436392adbdf6db096ab256d260f"} Jul 21 20:19:32.892429 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.892400 2567 scope.go:117] "RemoveContainer" containerID="fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae" Jul 21 20:19:32.893589 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.893564 2567 generic.go:356] "Generic (PLEG): container finished" podID="1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" containerID="c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3" exitCode=0 Jul 21 20:19:32.893661 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.893604 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bc45556cb-q9sn8" Jul 21 20:19:32.893661 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.893620 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc45556cb-q9sn8" event={"ID":"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47","Type":"ContainerDied","Data":"c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3"} Jul 21 20:19:32.893661 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.893646 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bc45556cb-q9sn8" event={"ID":"1e56abe8-3aa8-4bfd-87f9-faf5ff159d47","Type":"ContainerDied","Data":"e97087107dd31c057d493e23569567604394cc13bb32c210c277a896035b3c86"} Jul 21 20:19:32.900799 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.900785 2567 scope.go:117] "RemoveContainer" containerID="fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae" Jul 21 20:19:32.901100 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:19:32.901082 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae\": container with ID starting with fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae not found: ID does not exist" containerID="fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae" Jul 21 20:19:32.901184 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.901119 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae"} err="failed to get container status \"fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae\": rpc error: code = NotFound desc = could not find container \"fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae\": container with ID starting with fc51bd7fb1cf6fc56e5d47615e52d97900dac0caf1ee90b59c2cb09c0089d9ae not found: ID does not exist" Jul 21 20:19:32.901184 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.901156 2567 scope.go:117] "RemoveContainer" containerID="c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3" Jul 21 20:19:32.908543 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.908525 2567 scope.go:117] "RemoveContainer" containerID="c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3" Jul 21 20:19:32.908776 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:19:32.908759 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3\": container with ID starting with c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3 not found: ID does not exist" containerID="c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3" Jul 21 20:19:32.908820 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.908784 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3"} err="failed to get container status \"c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3\": rpc error: code = NotFound desc = could not find container \"c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3\": container with ID starting with c15b0a1c373be3a545f40c13f1f597c00df0670f5a9b9f5bd47024a3ab4ab4c3 not found: ID does not exist" Jul 21 20:19:32.910207 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.910190 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-oauth-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:32.910272 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.910207 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:32.910272 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.910222 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-service-ca\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:32.910272 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.910232 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:32.910272 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.910241 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-console-oauth-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:32.910272 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.910249 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdn99\" (UniqueName: \"kubernetes.io/projected/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47-kube-api-access-vdn99\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:32.914233 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.914212 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bc45556cb-q9sn8"] Jul 21 20:19:32.920037 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:32.920019 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bc45556cb-q9sn8"] Jul 21 20:19:33.010772 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.010696 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") pod \"7fda0385-d801-453a-ab06-bcac6b323a07\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " Jul 21 20:19:33.010772 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.010737 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrtnc\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-kube-api-access-mrtnc\") pod \"7fda0385-d801-453a-ab06-bcac6b323a07\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " Jul 21 20:19:33.010772 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.010766 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-image-registry-private-configuration\") pod \"7fda0385-d801-453a-ab06-bcac6b323a07\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " Jul 21 20:19:33.011075 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.010883 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-registry-certificates\") pod \"7fda0385-d801-453a-ab06-bcac6b323a07\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " Jul 21 20:19:33.011075 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.010931 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-bound-sa-token\") pod \"7fda0385-d801-453a-ab06-bcac6b323a07\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " Jul 21 20:19:33.011075 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.010993 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7fda0385-d801-453a-ab06-bcac6b323a07-ca-trust-extracted\") pod \"7fda0385-d801-453a-ab06-bcac6b323a07\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " Jul 21 20:19:33.011075 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.011013 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-installation-pull-secrets\") pod \"7fda0385-d801-453a-ab06-bcac6b323a07\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " Jul 21 20:19:33.011075 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.011061 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-trusted-ca\") pod \"7fda0385-d801-453a-ab06-bcac6b323a07\" (UID: \"7fda0385-d801-453a-ab06-bcac6b323a07\") " Jul 21 20:19:33.011643 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.011316 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7fda0385-d801-453a-ab06-bcac6b323a07" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:33.011643 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.011400 2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-registry-certificates\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:33.011806 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.011670 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7fda0385-d801-453a-ab06-bcac6b323a07" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:33.013279 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.013251 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-kube-api-access-mrtnc" (OuterVolumeSpecName: "kube-api-access-mrtnc") pod "7fda0385-d801-453a-ab06-bcac6b323a07" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07"). InnerVolumeSpecName "kube-api-access-mrtnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:19:33.013386 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.013305 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7fda0385-d801-453a-ab06-bcac6b323a07" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:19:33.013386 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.013371 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7fda0385-d801-453a-ab06-bcac6b323a07" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:19:33.013386 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.013377 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7fda0385-d801-453a-ab06-bcac6b323a07" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:19:33.013808 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.013788 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7fda0385-d801-453a-ab06-bcac6b323a07" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:19:33.019324 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.019302 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fda0385-d801-453a-ab06-bcac6b323a07-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7fda0385-d801-453a-ab06-bcac6b323a07" (UID: "7fda0385-d801-453a-ab06-bcac6b323a07"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:19:33.112264 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.112228 2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-registry-tls\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:33.112264 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.112258 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrtnc\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-kube-api-access-mrtnc\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:33.112264 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.112269 2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-image-registry-private-configuration\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:33.112455 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.112280 2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fda0385-d801-453a-ab06-bcac6b323a07-bound-sa-token\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:33.112455 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.112288 2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7fda0385-d801-453a-ab06-bcac6b323a07-ca-trust-extracted\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:33.112455 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.112296 2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7fda0385-d801-453a-ab06-bcac6b323a07-installation-pull-secrets\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:33.112455 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.112304 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7fda0385-d801-453a-ab06-bcac6b323a07-trusted-ca\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:33.215889 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.215864 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-664d77bf4-dsmcf"] Jul 21 20:19:33.220004 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.219975 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-664d77bf4-dsmcf"] Jul 21 20:19:33.804794 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.804721 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" path="/var/lib/kubelet/pods/1e56abe8-3aa8-4bfd-87f9-faf5ff159d47/volumes" Jul 21 20:19:33.805234 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:33.805139 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fda0385-d801-453a-ab06-bcac6b323a07" path="/var/lib/kubelet/pods/7fda0385-d801-453a-ab06-bcac6b323a07/volumes" Jul 21 20:19:34.906173 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:34.906139 2567 generic.go:356] "Generic (PLEG): container finished" podID="3b28e11a-e4ce-4b63-966d-60864726d9fc" containerID="afcb0f271e14fb5b2860b346361a5da9d4a7ec94a6984a803e231e62c03f53a8" exitCode=0 Jul 21 20:19:34.906624 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:34.906217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" event={"ID":"3b28e11a-e4ce-4b63-966d-60864726d9fc","Type":"ContainerDied","Data":"afcb0f271e14fb5b2860b346361a5da9d4a7ec94a6984a803e231e62c03f53a8"} Jul 21 20:19:34.906686 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:34.906658 2567 scope.go:117] "RemoveContainer" containerID="afcb0f271e14fb5b2860b346361a5da9d4a7ec94a6984a803e231e62c03f53a8" Jul 21 20:19:34.908027 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:34.908001 2567 generic.go:356] "Generic (PLEG): container finished" podID="4aec0aca-f346-46fb-8392-24e10c582da2" containerID="7e8b7569d69b2b1badc03e008c8209a84a7b6a18c2e7c8ab63573186c6bd18f5" exitCode=0 Jul 21 20:19:34.908081 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:34.908034 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-95dqt" event={"ID":"4aec0aca-f346-46fb-8392-24e10c582da2","Type":"ContainerDied","Data":"7e8b7569d69b2b1badc03e008c8209a84a7b6a18c2e7c8ab63573186c6bd18f5"} Jul 21 20:19:34.908371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:34.908352 2567 scope.go:117] "RemoveContainer" containerID="7e8b7569d69b2b1badc03e008c8209a84a7b6a18c2e7c8ab63573186c6bd18f5" Jul 21 20:19:35.914826 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:35.914789 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-66qg5" event={"ID":"3b28e11a-e4ce-4b63-966d-60864726d9fc","Type":"ContainerStarted","Data":"fceae634a38450410973e881ad11d3d25b4a301c9671bf339619572f6f65ab21"} Jul 21 20:19:35.916743 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:35.916715 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-95dqt" event={"ID":"4aec0aca-f346-46fb-8392-24e10c582da2","Type":"ContainerStarted","Data":"d2879b18f53dd58f6f6232e0744a6eb0e7cf6b7bb6dc11292d73a3719d0813d1"} Jul 21 20:19:37.826298 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:37.826248 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86cd5fbb68-s2d8r" podUID="11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" containerName="console" containerID="cri-o://4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a" gracePeriod=15 Jul 21 20:19:38.094738 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.094662 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:38.155353 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155320 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-trusted-ca-bundle\") pod \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " Jul 21 20:19:38.155518 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155370 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-oauth-config\") pod \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " Jul 21 20:19:38.155518 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155409 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-service-ca\") pod \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " Jul 21 20:19:38.155610 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155576 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-config\") pod \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " Jul 21 20:19:38.155664 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155627 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxkn5\" (UniqueName: \"kubernetes.io/projected/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-kube-api-access-dxkn5\") pod \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " Jul 21 20:19:38.155717 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155693 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-serving-cert\") pod \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " Jul 21 20:19:38.155770 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155724 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-oauth-serving-cert\") pod \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\" (UID: \"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b\") " Jul 21 20:19:38.155821 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155759 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-service-ca" (OuterVolumeSpecName: "service-ca") pod "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" (UID: "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:38.155821 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155774 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" (UID: "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:38.156020 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.155998 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-config" (OuterVolumeSpecName: "console-config") pod "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" (UID: "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:38.156020 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.156012 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-trusted-ca-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:38.156122 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.156029 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-service-ca\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:38.156122 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.156055 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" (UID: "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:38.157662 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.157634 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" (UID: "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:19:38.157778 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.157725 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-kube-api-access-dxkn5" (OuterVolumeSpecName: "kube-api-access-dxkn5") pod "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" (UID: "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b"). InnerVolumeSpecName "kube-api-access-dxkn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:19:38.157823 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.157768 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" (UID: "11c1fbd2-2af2-40d4-8d20-be78ba9bf89b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:19:38.257430 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.257389 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-oauth-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:38.257430 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.257420 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:38.257430 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.257430 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxkn5\" (UniqueName: \"kubernetes.io/projected/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-kube-api-access-dxkn5\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:38.257430 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.257438 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-console-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:38.257679 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.257448 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b-oauth-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:38.931206 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.931165 2567 generic.go:356] "Generic (PLEG): container finished" podID="11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" containerID="4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a" exitCode=0 Jul 21 20:19:38.931635 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.931216 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86cd5fbb68-s2d8r" Jul 21 20:19:38.931635 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.931214 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cd5fbb68-s2d8r" event={"ID":"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b","Type":"ContainerDied","Data":"4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a"} Jul 21 20:19:38.931635 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.931346 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86cd5fbb68-s2d8r" event={"ID":"11c1fbd2-2af2-40d4-8d20-be78ba9bf89b","Type":"ContainerDied","Data":"812102f4926d5e0e351e0d4988a8c2a54db1c584d5b0bfabb4f2bf864fce547b"} Jul 21 20:19:38.931635 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.931372 2567 scope.go:117] "RemoveContainer" containerID="4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a" Jul 21 20:19:38.940253 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.940239 2567 scope.go:117] "RemoveContainer" containerID="4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a" Jul 21 20:19:38.940483 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:19:38.940464 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a\": container with ID starting with 4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a not found: ID does not exist" containerID="4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a" Jul 21 20:19:38.940543 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.940491 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a"} err="failed to get container status \"4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a\": rpc error: code = NotFound desc = could not find container \"4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a\": container with ID starting with 4502a8ab5ca5b4e5b0d0a0a5ccb39415ec39feb112e25c9a519edaaf6ff8575a not found: ID does not exist" Jul 21 20:19:38.951824 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.951798 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86cd5fbb68-s2d8r"] Jul 21 20:19:38.954142 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:38.954122 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86cd5fbb68-s2d8r"] Jul 21 20:19:39.804720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:39.804688 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" path="/var/lib/kubelet/pods/11c1fbd2-2af2-40d4-8d20-be78ba9bf89b/volumes" Jul 21 20:19:40.941875 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:40.941843 2567 generic.go:356] "Generic (PLEG): container finished" podID="7de5f4f5-4560-4fd3-93b3-ebb043fe600f" containerID="1108d7f8d24233f23782659586bdfc1b6417d3c47c902e91f23d1c1427637877" exitCode=0 Jul 21 20:19:40.942276 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:40.941915 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" event={"ID":"7de5f4f5-4560-4fd3-93b3-ebb043fe600f","Type":"ContainerDied","Data":"1108d7f8d24233f23782659586bdfc1b6417d3c47c902e91f23d1c1427637877"} Jul 21 20:19:40.942276 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:40.942242 2567 scope.go:117] "RemoveContainer" containerID="1108d7f8d24233f23782659586bdfc1b6417d3c47c902e91f23d1c1427637877" Jul 21 20:19:41.948352 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:41.948320 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-f4b7v" event={"ID":"7de5f4f5-4560-4fd3-93b3-ebb043fe600f","Type":"ContainerStarted","Data":"2913330be8a8ec13f5c5fe46b055ffb286fd2dd720a948f1aa3fdc3dba985acd"} Jul 21 20:19:45.094609 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:45.094578 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:45.098411 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:45.098393 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54ff9d6fdf-gmm5p" Jul 21 20:19:47.922687 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:47.922623 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-784679754-lbq5j" podUID="66f61e22-e29b-404e-8e3c-e8d8c92c35cb" containerName="console" containerID="cri-o://9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b" gracePeriod=15 Jul 21 20:19:48.200819 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.200791 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:48.253609 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.253580 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-oauth-config\") pod \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " Jul 21 20:19:48.253759 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.253618 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-oauth-serving-cert\") pod \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " Jul 21 20:19:48.253759 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.253672 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt8sd\" (UniqueName: \"kubernetes.io/projected/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-kube-api-access-lt8sd\") pod \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " Jul 21 20:19:48.253759 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.253705 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-trusted-ca-bundle\") pod \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " Jul 21 20:19:48.253759 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.253728 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-serving-cert\") pod \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " Jul 21 20:19:48.254018 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.253796 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-config\") pod \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " Jul 21 20:19:48.254018 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.253836 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-service-ca\") pod \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\" (UID: \"66f61e22-e29b-404e-8e3c-e8d8c92c35cb\") " Jul 21 20:19:48.254133 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.254071 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66f61e22-e29b-404e-8e3c-e8d8c92c35cb" (UID: "66f61e22-e29b-404e-8e3c-e8d8c92c35cb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:48.254133 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.254099 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "66f61e22-e29b-404e-8e3c-e8d8c92c35cb" (UID: "66f61e22-e29b-404e-8e3c-e8d8c92c35cb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:48.254240 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.254154 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-oauth-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:48.254393 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.254368 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-config" (OuterVolumeSpecName: "console-config") pod "66f61e22-e29b-404e-8e3c-e8d8c92c35cb" (UID: "66f61e22-e29b-404e-8e3c-e8d8c92c35cb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:48.254393 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.254382 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-service-ca" (OuterVolumeSpecName: "service-ca") pod "66f61e22-e29b-404e-8e3c-e8d8c92c35cb" (UID: "66f61e22-e29b-404e-8e3c-e8d8c92c35cb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:19:48.256124 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.256097 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66f61e22-e29b-404e-8e3c-e8d8c92c35cb" (UID: "66f61e22-e29b-404e-8e3c-e8d8c92c35cb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:19:48.256215 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.256184 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-kube-api-access-lt8sd" (OuterVolumeSpecName: "kube-api-access-lt8sd") pod "66f61e22-e29b-404e-8e3c-e8d8c92c35cb" (UID: "66f61e22-e29b-404e-8e3c-e8d8c92c35cb"). InnerVolumeSpecName "kube-api-access-lt8sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:19:48.256310 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.256289 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66f61e22-e29b-404e-8e3c-e8d8c92c35cb" (UID: "66f61e22-e29b-404e-8e3c-e8d8c92c35cb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:19:48.354917 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.354884 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:48.354917 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.354911 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-service-ca\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:48.354917 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.354920 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-oauth-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:48.354917 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.354929 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8sd\" (UniqueName: \"kubernetes.io/projected/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-kube-api-access-lt8sd\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:48.355260 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.354937 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-trusted-ca-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:48.355260 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.354986 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66f61e22-e29b-404e-8e3c-e8d8c92c35cb-console-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:19:48.982972 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.982908 2567 generic.go:356] "Generic (PLEG): container finished" podID="66f61e22-e29b-404e-8e3c-e8d8c92c35cb" containerID="9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b" exitCode=0 Jul 21 20:19:48.982972 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.982970 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784679754-lbq5j" Jul 21 20:19:48.983448 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.982990 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784679754-lbq5j" event={"ID":"66f61e22-e29b-404e-8e3c-e8d8c92c35cb","Type":"ContainerDied","Data":"9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b"} Jul 21 20:19:48.983448 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.983046 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784679754-lbq5j" event={"ID":"66f61e22-e29b-404e-8e3c-e8d8c92c35cb","Type":"ContainerDied","Data":"e915ce6930c5bfbeb34f923109ffa8612192a182c94a0f92aa744af5fa247124"} Jul 21 20:19:48.983448 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.983077 2567 scope.go:117] "RemoveContainer" containerID="9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b" Jul 21 20:19:48.993850 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.993761 2567 scope.go:117] "RemoveContainer" containerID="9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b" Jul 21 20:19:48.994331 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:19:48.994252 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b\": container with ID starting with 9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b not found: ID does not exist" containerID="9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b" Jul 21 20:19:48.994331 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:48.994294 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b"} err="failed to get container status \"9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b\": rpc error: code = NotFound desc = could not find container \"9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b\": container with ID starting with 9c92615e91579d9904d6cd197769814ca4d5ac77da3368b2841ea4ac90c7866b not found: ID does not exist" Jul 21 20:19:49.008086 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:49.008040 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-784679754-lbq5j"] Jul 21 20:19:49.011991 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:49.011970 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-784679754-lbq5j"] Jul 21 20:19:49.805508 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:19:49.805471 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f61e22-e29b-404e-8e3c-e8d8c92c35cb" path="/var/lib/kubelet/pods/66f61e22-e29b-404e-8e3c-e8d8c92c35cb/volumes" Jul 21 20:20:19.635833 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:19.635785 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:20:19.638114 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:19.638093 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/011ea63a-4a29-4f3e-9710-4b9b299ba69c-metrics-certs\") pod \"network-metrics-daemon-f57lq\" (UID: \"011ea63a-4a29-4f3e-9710-4b9b299ba69c\") " pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:20:19.702033 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:19.702004 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bxdt9\"" Jul 21 20:20:19.710082 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:19.710059 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f57lq" Jul 21 20:20:19.861446 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:19.861414 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f57lq"] Jul 21 20:20:19.866845 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:20:19.866809 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011ea63a_4a29_4f3e_9710_4b9b299ba69c.slice/crio-1d7281b3d11d74c5c335bfa1088e1ce9c7d5f283dda973b01b880a8f6146fd04 WatchSource:0}: Error finding container 1d7281b3d11d74c5c335bfa1088e1ce9c7d5f283dda973b01b880a8f6146fd04: Status 404 returned error can't find the container with id 1d7281b3d11d74c5c335bfa1088e1ce9c7d5f283dda973b01b880a8f6146fd04 Jul 21 20:20:20.111786 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:20.111706 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f57lq" event={"ID":"011ea63a-4a29-4f3e-9710-4b9b299ba69c","Type":"ContainerStarted","Data":"1d7281b3d11d74c5c335bfa1088e1ce9c7d5f283dda973b01b880a8f6146fd04"} Jul 21 20:20:21.117219 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:21.117190 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f57lq" event={"ID":"011ea63a-4a29-4f3e-9710-4b9b299ba69c","Type":"ContainerStarted","Data":"b3d8cd9da7c9d493b30a345bfb5c69d83dd9aea932a5af974a4aaf73b9f66564"} Jul 21 20:20:21.117691 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:21.117242 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f57lq" event={"ID":"011ea63a-4a29-4f3e-9710-4b9b299ba69c","Type":"ContainerStarted","Data":"1aee97cf49bede0c432e5dda76a8a2045f4e45d6be68d3b105e7d35edfe21026"} Jul 21 20:20:21.134533 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:21.134476 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f57lq" podStartSLOduration=375.249012409 podStartE2EDuration="6m16.134455938s" totalImagesPullingTime="885.443529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:20:21.131841902 +0000 UTC m=+375.811968349" watchObservedRunningTime="2026-07-21 20:20:21.134455938 +0000 UTC m=+375.814582385" Jul 21 20:20:31.332665 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.332629 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-679fd5ff7b-fkhgj"] Jul 21 20:20:31.333077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.332980 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" containerName="console" Jul 21 20:20:31.333077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.332993 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" containerName="console" Jul 21 20:20:31.333077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333013 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fda0385-d801-453a-ab06-bcac6b323a07" containerName="registry" Jul 21 20:20:31.333077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333019 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fda0385-d801-453a-ab06-bcac6b323a07" containerName="registry" Jul 21 20:20:31.333077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333027 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" containerName="console" Jul 21 20:20:31.333077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333032 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" containerName="console" Jul 21 20:20:31.333077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333039 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f61e22-e29b-404e-8e3c-e8d8c92c35cb" containerName="console" Jul 21 20:20:31.333077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333045 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f61e22-e29b-404e-8e3c-e8d8c92c35cb" containerName="console" Jul 21 20:20:31.333363 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333096 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f61e22-e29b-404e-8e3c-e8d8c92c35cb" containerName="console" Jul 21 20:20:31.333363 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333103 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e56abe8-3aa8-4bfd-87f9-faf5ff159d47" containerName="console" Jul 21 20:20:31.333363 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333110 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c1fbd2-2af2-40d4-8d20-be78ba9bf89b" containerName="console" Jul 21 20:20:31.333363 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.333118 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fda0385-d801-453a-ab06-bcac6b323a07" containerName="registry" Jul 21 20:20:31.336071 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.336055 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.350865 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.350838 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679fd5ff7b-fkhgj"] Jul 21 20:20:31.434186 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.434143 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-serving-cert\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.434377 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.434197 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-oauth-config\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.434377 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.434293 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-config\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.434377 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.434371 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj89r\" (UniqueName: \"kubernetes.io/projected/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-kube-api-access-vj89r\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.434553 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.434415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-oauth-serving-cert\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.434553 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.434445 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-trusted-ca-bundle\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.434553 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.434479 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-service-ca\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.534860 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.534824 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-oauth-config\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535085 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.534871 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-config\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535085 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.534917 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj89r\" (UniqueName: \"kubernetes.io/projected/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-kube-api-access-vj89r\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535085 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.534978 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-oauth-serving-cert\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535085 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.535007 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-trusted-ca-bundle\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535085 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.535033 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-service-ca\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535085 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.535063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-serving-cert\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535736 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.535708 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-config\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535844 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.535745 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-service-ca\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535844 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.535750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-oauth-serving-cert\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.535973 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.535892 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-trusted-ca-bundle\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.537396 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.537378 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-oauth-config\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.537484 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.537468 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-serving-cert\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.542751 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.542730 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj89r\" (UniqueName: \"kubernetes.io/projected/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-kube-api-access-vj89r\") pod \"console-679fd5ff7b-fkhgj\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.645370 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.645334 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:31.789108 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:31.789079 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-679fd5ff7b-fkhgj"] Jul 21 20:20:31.792295 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:20:31.792266 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3002ab_abb1_4d70_ab45_dcd1bd4d74a2.slice/crio-1aefd698d0ff08f53f99efcbe8358ed498fb31338ac6730620569e33064f59cb WatchSource:0}: Error finding container 1aefd698d0ff08f53f99efcbe8358ed498fb31338ac6730620569e33064f59cb: Status 404 returned error can't find the container with id 1aefd698d0ff08f53f99efcbe8358ed498fb31338ac6730620569e33064f59cb Jul 21 20:20:32.162515 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:32.162482 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679fd5ff7b-fkhgj" event={"ID":"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2","Type":"ContainerStarted","Data":"e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4"} Jul 21 20:20:32.162692 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:32.162533 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679fd5ff7b-fkhgj" event={"ID":"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2","Type":"ContainerStarted","Data":"1aefd698d0ff08f53f99efcbe8358ed498fb31338ac6730620569e33064f59cb"} Jul 21 20:20:32.182245 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:32.182193 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-679fd5ff7b-fkhgj" podStartSLOduration=1.182176881 podStartE2EDuration="1.182176881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:20:32.180617509 +0000 UTC m=+386.860743978" watchObservedRunningTime="2026-07-21 20:20:32.182176881 +0000 UTC m=+386.862303317" Jul 21 20:20:41.646448 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:41.646412 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:41.646897 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:41.646460 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:41.651228 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:41.651206 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:42.209497 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:42.209472 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:20:42.259685 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:20:42.259653 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b6c546f8-bkz24"] Jul 21 20:21:07.279968 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.279889 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66b6c546f8-bkz24" podUID="289eed1f-113e-4061-b4ba-1f8fb0e1ba40" containerName="console" containerID="cri-o://d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d" gracePeriod=15 Jul 21 20:21:07.527520 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.527493 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:21:07.547306 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547243 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-trusted-ca-bundle\") pod \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " Jul 21 20:21:07.547306 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547274 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-serving-cert\") pod \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " Jul 21 20:21:07.547478 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547309 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-service-ca\") pod \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " Jul 21 20:21:07.547478 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547459 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-config\") pod \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " Jul 21 20:21:07.547581 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547509 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-oauth-config\") pod \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " Jul 21 20:21:07.547581 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547548 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtn4t\" (UniqueName: \"kubernetes.io/projected/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-kube-api-access-mtn4t\") pod \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " Jul 21 20:21:07.547680 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547611 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-oauth-serving-cert\") pod \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\" (UID: \"289eed1f-113e-4061-b4ba-1f8fb0e1ba40\") " Jul 21 20:21:07.547680 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547622 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "289eed1f-113e-4061-b4ba-1f8fb0e1ba40" (UID: "289eed1f-113e-4061-b4ba-1f8fb0e1ba40"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:21:07.547769 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547648 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-service-ca" (OuterVolumeSpecName: "service-ca") pod "289eed1f-113e-4061-b4ba-1f8fb0e1ba40" (UID: "289eed1f-113e-4061-b4ba-1f8fb0e1ba40"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:21:07.547848 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547810 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-config" (OuterVolumeSpecName: "console-config") pod "289eed1f-113e-4061-b4ba-1f8fb0e1ba40" (UID: "289eed1f-113e-4061-b4ba-1f8fb0e1ba40"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:21:07.547934 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547875 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-trusted-ca-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:07.547934 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547896 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-service-ca\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:07.547934 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.547909 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:07.548073 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.548048 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "289eed1f-113e-4061-b4ba-1f8fb0e1ba40" (UID: "289eed1f-113e-4061-b4ba-1f8fb0e1ba40"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:21:07.549616 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.549588 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-kube-api-access-mtn4t" (OuterVolumeSpecName: "kube-api-access-mtn4t") pod "289eed1f-113e-4061-b4ba-1f8fb0e1ba40" (UID: "289eed1f-113e-4061-b4ba-1f8fb0e1ba40"). InnerVolumeSpecName "kube-api-access-mtn4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:21:07.549616 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.549592 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "289eed1f-113e-4061-b4ba-1f8fb0e1ba40" (UID: "289eed1f-113e-4061-b4ba-1f8fb0e1ba40"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:21:07.549748 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.549654 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "289eed1f-113e-4061-b4ba-1f8fb0e1ba40" (UID: "289eed1f-113e-4061-b4ba-1f8fb0e1ba40"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:21:07.648622 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.648589 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-oauth-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:07.648622 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.648620 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtn4t\" (UniqueName: \"kubernetes.io/projected/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-kube-api-access-mtn4t\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:07.648622 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.648629 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-oauth-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:07.648855 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:07.648637 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/289eed1f-113e-4061-b4ba-1f8fb0e1ba40-console-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:08.318490 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:08.318455 2567 generic.go:356] "Generic (PLEG): container finished" podID="289eed1f-113e-4061-b4ba-1f8fb0e1ba40" containerID="d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d" exitCode=0 Jul 21 20:21:08.318930 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:08.318500 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b6c546f8-bkz24" Jul 21 20:21:08.318930 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:08.318539 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b6c546f8-bkz24" event={"ID":"289eed1f-113e-4061-b4ba-1f8fb0e1ba40","Type":"ContainerDied","Data":"d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d"} Jul 21 20:21:08.318930 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:08.318602 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b6c546f8-bkz24" event={"ID":"289eed1f-113e-4061-b4ba-1f8fb0e1ba40","Type":"ContainerDied","Data":"0cf7d21a7a9d7928e6c389cbff77a5ef47b492b14abea0815d6161aa47bce18a"} Jul 21 20:21:08.318930 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:08.318622 2567 scope.go:117] "RemoveContainer" containerID="d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d" Jul 21 20:21:08.326920 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:08.326900 2567 scope.go:117] "RemoveContainer" containerID="d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d" Jul 21 20:21:08.327205 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:21:08.327187 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d\": container with ID starting with d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d not found: ID does not exist" containerID="d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d" Jul 21 20:21:08.327257 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:08.327213 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d"} err="failed to get container status \"d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d\": rpc error: code = NotFound desc = could not find container \"d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d\": container with ID starting with d3aa9238848ccba4fbd9898a1331c89a35289a3aedac3d98d5ce4ddd4d99e01d not found: ID does not exist" Jul 21 20:21:08.338667 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:08.338643 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b6c546f8-bkz24"] Jul 21 20:21:08.340346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:08.340327 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66b6c546f8-bkz24"] Jul 21 20:21:09.803678 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:09.803644 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289eed1f-113e-4061-b4ba-1f8fb0e1ba40" path="/var/lib/kubelet/pods/289eed1f-113e-4061-b4ba-1f8fb0e1ba40/volumes" Jul 21 20:21:16.052438 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.052401 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc"] Jul 21 20:21:16.053022 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.052876 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289eed1f-113e-4061-b4ba-1f8fb0e1ba40" containerName="console" Jul 21 20:21:16.053022 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.052896 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="289eed1f-113e-4061-b4ba-1f8fb0e1ba40" containerName="console" Jul 21 20:21:16.053022 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.053001 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="289eed1f-113e-4061-b4ba-1f8fb0e1ba40" containerName="console" Jul 21 20:21:16.057584 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.057563 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.060336 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.060316 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 20:21:16.060467 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.060316 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fsfl2\"" Jul 21 20:21:16.060538 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.060399 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 20:21:16.068697 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.068647 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc"] Jul 21 20:21:16.116657 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.116621 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.116809 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.116703 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nr5f\" (UniqueName: \"kubernetes.io/projected/11b90618-5538-45ee-aaae-d0ba72e2b35f-kube-api-access-6nr5f\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.116809 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.116727 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.217766 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.217715 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nr5f\" (UniqueName: \"kubernetes.io/projected/11b90618-5538-45ee-aaae-d0ba72e2b35f-kube-api-access-6nr5f\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.218003 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.217776 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.218003 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.217802 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.218201 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.218180 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.218272 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.218231 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.227097 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.227077 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nr5f\" (UniqueName: \"kubernetes.io/projected/11b90618-5538-45ee-aaae-d0ba72e2b35f-kube-api-access-6nr5f\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.366566 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.366484 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:16.495504 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:16.495482 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc"] Jul 21 20:21:16.497741 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:21:16.497716 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b90618_5538_45ee_aaae_d0ba72e2b35f.slice/crio-abe0157f0a1ec02eef2dbd23a141ac90a0e432a3a52e08678d416e52da53e1e6 WatchSource:0}: Error finding container abe0157f0a1ec02eef2dbd23a141ac90a0e432a3a52e08678d416e52da53e1e6: Status 404 returned error can't find the container with id abe0157f0a1ec02eef2dbd23a141ac90a0e432a3a52e08678d416e52da53e1e6 Jul 21 20:21:17.358981 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:17.358925 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" event={"ID":"11b90618-5538-45ee-aaae-d0ba72e2b35f","Type":"ContainerStarted","Data":"abe0157f0a1ec02eef2dbd23a141ac90a0e432a3a52e08678d416e52da53e1e6"} Jul 21 20:21:23.388088 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:23.388032 2567 generic.go:356] "Generic (PLEG): container finished" podID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerID="5cbe47321dd0684d01672df598f46404a2e7f20b754fadccd463b4d9bc264b56" exitCode=0 Jul 21 20:21:23.388461 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:23.388116 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" event={"ID":"11b90618-5538-45ee-aaae-d0ba72e2b35f","Type":"ContainerDied","Data":"5cbe47321dd0684d01672df598f46404a2e7f20b754fadccd463b4d9bc264b56"} Jul 21 20:21:26.404494 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:26.404463 2567 generic.go:356] "Generic (PLEG): container finished" podID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerID="2442804476afbfc7d9157ba145f42d6d8f419d9d0c9f68008f531c3509a40e32" exitCode=0 Jul 21 20:21:26.404908 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:26.404560 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" event={"ID":"11b90618-5538-45ee-aaae-d0ba72e2b35f","Type":"ContainerDied","Data":"2442804476afbfc7d9157ba145f42d6d8f419d9d0c9f68008f531c3509a40e32"} Jul 21 20:21:35.450288 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:35.450248 2567 generic.go:356] "Generic (PLEG): container finished" podID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerID="cb31ee88d129ce1e8fe474475b23e077477594743f52b25d387a8df3dd9d383a" exitCode=0 Jul 21 20:21:35.450667 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:35.450332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" event={"ID":"11b90618-5538-45ee-aaae-d0ba72e2b35f","Type":"ContainerDied","Data":"cb31ee88d129ce1e8fe474475b23e077477594743f52b25d387a8df3dd9d383a"} Jul 21 20:21:36.588534 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.588512 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:36.704162 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.704131 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nr5f\" (UniqueName: \"kubernetes.io/projected/11b90618-5538-45ee-aaae-d0ba72e2b35f-kube-api-access-6nr5f\") pod \"11b90618-5538-45ee-aaae-d0ba72e2b35f\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " Jul 21 20:21:36.704323 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.704197 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-util\") pod \"11b90618-5538-45ee-aaae-d0ba72e2b35f\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " Jul 21 20:21:36.704385 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.704365 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-bundle\") pod \"11b90618-5538-45ee-aaae-d0ba72e2b35f\" (UID: \"11b90618-5538-45ee-aaae-d0ba72e2b35f\") " Jul 21 20:21:36.705016 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.704990 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-bundle" (OuterVolumeSpecName: "bundle") pod "11b90618-5538-45ee-aaae-d0ba72e2b35f" (UID: "11b90618-5538-45ee-aaae-d0ba72e2b35f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:21:36.706335 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.706276 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b90618-5538-45ee-aaae-d0ba72e2b35f-kube-api-access-6nr5f" (OuterVolumeSpecName: "kube-api-access-6nr5f") pod "11b90618-5538-45ee-aaae-d0ba72e2b35f" (UID: "11b90618-5538-45ee-aaae-d0ba72e2b35f"). InnerVolumeSpecName "kube-api-access-6nr5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:21:36.709529 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.709508 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-util" (OuterVolumeSpecName: "util") pod "11b90618-5538-45ee-aaae-d0ba72e2b35f" (UID: "11b90618-5538-45ee-aaae-d0ba72e2b35f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:21:36.806006 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.805963 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-util\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:36.806006 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.806000 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11b90618-5538-45ee-aaae-d0ba72e2b35f-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:36.806006 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:36.806011 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nr5f\" (UniqueName: \"kubernetes.io/projected/11b90618-5538-45ee-aaae-d0ba72e2b35f-kube-api-access-6nr5f\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:21:37.460484 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:37.460455 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" Jul 21 20:21:37.460705 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:37.460454 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bs4xc" event={"ID":"11b90618-5538-45ee-aaae-d0ba72e2b35f","Type":"ContainerDied","Data":"abe0157f0a1ec02eef2dbd23a141ac90a0e432a3a52e08678d416e52da53e1e6"} Jul 21 20:21:37.460705 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:37.460572 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abe0157f0a1ec02eef2dbd23a141ac90a0e432a3a52e08678d416e52da53e1e6" Jul 21 20:21:44.499038 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.499007 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92"] Jul 21 20:21:44.499493 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.499300 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerName="util" Jul 21 20:21:44.499493 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.499311 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerName="util" Jul 21 20:21:44.499493 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.499325 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerName="pull" Jul 21 20:21:44.499493 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.499331 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerName="pull" Jul 21 20:21:44.499493 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.499344 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerName="extract" Jul 21 20:21:44.499493 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.499350 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerName="extract" Jul 21 20:21:44.499493 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.499404 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="11b90618-5538-45ee-aaae-d0ba72e2b35f" containerName="extract" Jul 21 20:21:44.503197 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.503182 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" Jul 21 20:21:44.505894 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.505870 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lc4zp\"" Jul 21 20:21:44.506080 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.506060 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 21 20:21:44.506138 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.506097 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 21 20:21:44.520308 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.520287 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92"] Jul 21 20:21:44.676502 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.676468 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrp4\" (UniqueName: \"kubernetes.io/projected/21853828-42c3-45f9-9214-05ad089ad0dd-kube-api-access-6nrp4\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4n92\" (UID: \"21853828-42c3-45f9-9214-05ad089ad0dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" Jul 21 20:21:44.676679 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.676542 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21853828-42c3-45f9-9214-05ad089ad0dd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4n92\" (UID: \"21853828-42c3-45f9-9214-05ad089ad0dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" Jul 21 20:21:44.777686 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.777604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrp4\" (UniqueName: \"kubernetes.io/projected/21853828-42c3-45f9-9214-05ad089ad0dd-kube-api-access-6nrp4\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4n92\" (UID: \"21853828-42c3-45f9-9214-05ad089ad0dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" Jul 21 20:21:44.777836 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.777686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21853828-42c3-45f9-9214-05ad089ad0dd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4n92\" (UID: \"21853828-42c3-45f9-9214-05ad089ad0dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" Jul 21 20:21:44.778152 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.778131 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21853828-42c3-45f9-9214-05ad089ad0dd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4n92\" (UID: \"21853828-42c3-45f9-9214-05ad089ad0dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" Jul 21 20:21:44.786459 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.786435 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrp4\" (UniqueName: \"kubernetes.io/projected/21853828-42c3-45f9-9214-05ad089ad0dd-kube-api-access-6nrp4\") pod \"cert-manager-operator-controller-manager-55c444bf86-x4n92\" (UID: \"21853828-42c3-45f9-9214-05ad089ad0dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" Jul 21 20:21:44.812025 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.811988 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" Jul 21 20:21:44.951510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:44.951450 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92"] Jul 21 20:21:44.954585 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:21:44.954557 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21853828_42c3_45f9_9214_05ad089ad0dd.slice/crio-9990e05a295aecccc7d4318557b4354ce6ea4831bad3208aba1fbcf921efe61d WatchSource:0}: Error finding container 9990e05a295aecccc7d4318557b4354ce6ea4831bad3208aba1fbcf921efe61d: Status 404 returned error can't find the container with id 9990e05a295aecccc7d4318557b4354ce6ea4831bad3208aba1fbcf921efe61d Jul 21 20:21:45.497545 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:45.497513 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" event={"ID":"21853828-42c3-45f9-9214-05ad089ad0dd","Type":"ContainerStarted","Data":"9990e05a295aecccc7d4318557b4354ce6ea4831bad3208aba1fbcf921efe61d"} Jul 21 20:21:47.508533 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.508500 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" event={"ID":"21853828-42c3-45f9-9214-05ad089ad0dd","Type":"ContainerStarted","Data":"74f8fcf0eaade18889c767eabf38417e7f321e7292470f152a37a8fc0e7d1912"} Jul 21 20:21:47.557076 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.557022 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x4n92" podStartSLOduration=1.873902073 podStartE2EDuration="3.557005154s" totalImagesPullingTime="1.683103081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:21:47.554424555 +0000 UTC m=+462.234551024" watchObservedRunningTime="2026-07-21 20:21:47.557005154 +0000 UTC m=+462.237131590" Jul 21 20:21:47.668303 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.668267 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f454h"] Jul 21 20:21:47.671730 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.671713 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:21:47.674063 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.674041 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 20:21:47.674159 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.674047 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 20:21:47.685778 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.685756 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f454h"] Jul 21 20:21:47.806685 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.806613 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ftm\" (UniqueName: \"kubernetes.io/projected/653390a0-691e-4f78-8e93-2b239dc23893-kube-api-access-x2ftm\") pod \"cert-manager-webhook-6486ffd997-f454h\" (UID: \"653390a0-691e-4f78-8e93-2b239dc23893\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:21:47.806685 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.806661 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/653390a0-691e-4f78-8e93-2b239dc23893-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f454h\" (UID: \"653390a0-691e-4f78-8e93-2b239dc23893\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:21:47.907364 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.907330 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ftm\" (UniqueName: \"kubernetes.io/projected/653390a0-691e-4f78-8e93-2b239dc23893-kube-api-access-x2ftm\") pod \"cert-manager-webhook-6486ffd997-f454h\" (UID: \"653390a0-691e-4f78-8e93-2b239dc23893\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:21:47.907550 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.907387 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/653390a0-691e-4f78-8e93-2b239dc23893-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f454h\" (UID: \"653390a0-691e-4f78-8e93-2b239dc23893\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:21:47.924031 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.924005 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/653390a0-691e-4f78-8e93-2b239dc23893-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-f454h\" (UID: \"653390a0-691e-4f78-8e93-2b239dc23893\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:21:47.924140 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.924130 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ftm\" (UniqueName: \"kubernetes.io/projected/653390a0-691e-4f78-8e93-2b239dc23893-kube-api-access-x2ftm\") pod \"cert-manager-webhook-6486ffd997-f454h\" (UID: \"653390a0-691e-4f78-8e93-2b239dc23893\") " pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:21:47.989216 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:47.989187 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:21:48.139340 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.139307 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-f454h"] Jul 21 20:21:48.142422 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:21:48.142387 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod653390a0_691e_4f78_8e93_2b239dc23893.slice/crio-56654d722e3fb86aa1aa677a132c61f482915962127889b3e292d0e05bf00d5b WatchSource:0}: Error finding container 56654d722e3fb86aa1aa677a132c61f482915962127889b3e292d0e05bf00d5b: Status 404 returned error can't find the container with id 56654d722e3fb86aa1aa677a132c61f482915962127889b3e292d0e05bf00d5b Jul 21 20:21:48.513748 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.513714 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" event={"ID":"653390a0-691e-4f78-8e93-2b239dc23893","Type":"ContainerStarted","Data":"56654d722e3fb86aa1aa677a132c61f482915962127889b3e292d0e05bf00d5b"} Jul 21 20:21:48.524209 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.524182 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7"] Jul 21 20:21:48.528634 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.528617 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.531237 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.531216 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 20:21:48.531366 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.531348 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 20:21:48.531990 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.531976 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fsfl2\"" Jul 21 20:21:48.551396 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.551373 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7"] Jul 21 20:21:48.713819 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.713791 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrf7n\" (UniqueName: \"kubernetes.io/projected/79a5cc8f-26cb-4ed9-b0d4-a11717742808-kube-api-access-vrf7n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.714025 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.713840 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.714025 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.713965 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.815204 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.815110 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrf7n\" (UniqueName: \"kubernetes.io/projected/79a5cc8f-26cb-4ed9-b0d4-a11717742808-kube-api-access-vrf7n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.815204 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.815150 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.815433 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.815216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.815651 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.815631 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.815720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.815705 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.829163 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.829145 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrf7n\" (UniqueName: \"kubernetes.io/projected/79a5cc8f-26cb-4ed9-b0d4-a11717742808-kube-api-access-vrf7n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.837992 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.837975 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:21:48.981454 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:48.981328 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7"] Jul 21 20:21:48.983933 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:21:48.983906 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a5cc8f_26cb_4ed9_b0d4_a11717742808.slice/crio-0b98154017f7b0a01ab901a27fca84a1a508733e516545a0849de20e3872b5d1 WatchSource:0}: Error finding container 0b98154017f7b0a01ab901a27fca84a1a508733e516545a0849de20e3872b5d1: Status 404 returned error can't find the container with id 0b98154017f7b0a01ab901a27fca84a1a508733e516545a0849de20e3872b5d1 Jul 21 20:21:49.520709 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:49.520673 2567 generic.go:356] "Generic (PLEG): container finished" podID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerID="1fc8d389f5cd378db901f398daca9966ea2119076953bcfc66cd4931b4bdcca4" exitCode=0 Jul 21 20:21:49.521185 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:49.520805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" event={"ID":"79a5cc8f-26cb-4ed9-b0d4-a11717742808","Type":"ContainerDied","Data":"1fc8d389f5cd378db901f398daca9966ea2119076953bcfc66cd4931b4bdcca4"} Jul 21 20:21:49.521185 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:49.520856 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" event={"ID":"79a5cc8f-26cb-4ed9-b0d4-a11717742808","Type":"ContainerStarted","Data":"0b98154017f7b0a01ab901a27fca84a1a508733e516545a0849de20e3872b5d1"} Jul 21 20:21:50.076744 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.076707 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k2p84"] Jul 21 20:21:50.080506 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.080479 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" Jul 21 20:21:50.083529 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.083508 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7mxkb\"" Jul 21 20:21:50.091653 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.091629 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k2p84"] Jul 21 20:21:50.228783 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.228747 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8de1a238-4fa6-438a-a885-d67fa780923a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k2p84\" (UID: \"8de1a238-4fa6-438a-a885-d67fa780923a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" Jul 21 20:21:50.228997 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.228850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp27s\" (UniqueName: \"kubernetes.io/projected/8de1a238-4fa6-438a-a885-d67fa780923a-kube-api-access-rp27s\") pod \"cert-manager-cainjector-559cdd8c87-k2p84\" (UID: \"8de1a238-4fa6-438a-a885-d67fa780923a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" Jul 21 20:21:50.329356 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.329272 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8de1a238-4fa6-438a-a885-d67fa780923a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k2p84\" (UID: \"8de1a238-4fa6-438a-a885-d67fa780923a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" Jul 21 20:21:50.329356 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.329341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp27s\" (UniqueName: \"kubernetes.io/projected/8de1a238-4fa6-438a-a885-d67fa780923a-kube-api-access-rp27s\") pod \"cert-manager-cainjector-559cdd8c87-k2p84\" (UID: \"8de1a238-4fa6-438a-a885-d67fa780923a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" Jul 21 20:21:50.338873 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.338851 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8de1a238-4fa6-438a-a885-d67fa780923a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k2p84\" (UID: \"8de1a238-4fa6-438a-a885-d67fa780923a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" Jul 21 20:21:50.339261 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.339236 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp27s\" (UniqueName: \"kubernetes.io/projected/8de1a238-4fa6-438a-a885-d67fa780923a-kube-api-access-rp27s\") pod \"cert-manager-cainjector-559cdd8c87-k2p84\" (UID: \"8de1a238-4fa6-438a-a885-d67fa780923a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" Jul 21 20:21:50.391482 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.391447 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" Jul 21 20:21:50.959836 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:50.959811 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k2p84"] Jul 21 20:21:50.962408 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:21:50.962384 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8de1a238_4fa6_438a_a885_d67fa780923a.slice/crio-5d823ab573010cfb5edf1578b353c781bc230c55091abb607a2d134125bd8cb8 WatchSource:0}: Error finding container 5d823ab573010cfb5edf1578b353c781bc230c55091abb607a2d134125bd8cb8: Status 404 returned error can't find the container with id 5d823ab573010cfb5edf1578b353c781bc230c55091abb607a2d134125bd8cb8 Jul 21 20:21:51.532249 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:51.532216 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" event={"ID":"653390a0-691e-4f78-8e93-2b239dc23893","Type":"ContainerStarted","Data":"aaf8622c94da8154132f1ffbfe9a3322965253ac15feecb809842d39410ce522"} Jul 21 20:21:51.532462 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:51.532288 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:21:51.533744 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:51.533727 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" event={"ID":"8de1a238-4fa6-438a-a885-d67fa780923a","Type":"ContainerStarted","Data":"b2c9819864916162f50c8249a9463b032dc8977f2ceb6fca29fbf4ca31e2e0e3"} Jul 21 20:21:51.533847 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:51.533751 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" event={"ID":"8de1a238-4fa6-438a-a885-d67fa780923a","Type":"ContainerStarted","Data":"5d823ab573010cfb5edf1578b353c781bc230c55091abb607a2d134125bd8cb8"} Jul 21 20:21:51.550713 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:51.550670 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" podStartSLOduration=1.8010908570000002 podStartE2EDuration="4.550656723s" totalImagesPullingTime="2.749565866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:21:51.549034185 +0000 UTC m=+466.229160639" watchObservedRunningTime="2026-07-21 20:21:51.550656723 +0000 UTC m=+466.230783161" Jul 21 20:21:51.565724 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:51.565685 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k2p84" podStartSLOduration=1.5656723750000001 podStartE2EDuration="1.565672375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:21:51.564317774 +0000 UTC m=+466.244444222" watchObservedRunningTime="2026-07-21 20:21:51.565672375 +0000 UTC m=+466.245798811" Jul 21 20:21:57.540818 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:21:57.540784 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-f454h" Jul 21 20:22:08.618919 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:08.618879 2567 generic.go:356] "Generic (PLEG): container finished" podID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerID="8a09bf288e2a806a3cc88b741a120e8fc37247784f662c12dad75775c9c90d20" exitCode=0 Jul 21 20:22:08.619338 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:08.618970 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" event={"ID":"79a5cc8f-26cb-4ed9-b0d4-a11717742808","Type":"ContainerDied","Data":"8a09bf288e2a806a3cc88b741a120e8fc37247784f662c12dad75775c9c90d20"} Jul 21 20:22:09.626176 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:09.626144 2567 generic.go:356] "Generic (PLEG): container finished" podID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerID="1f8c1f86d7cf663f296e209b57054cd1177014a3b2e96ee4a6b54c20234a4129" exitCode=0 Jul 21 20:22:09.626555 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:09.626231 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" event={"ID":"79a5cc8f-26cb-4ed9-b0d4-a11717742808","Type":"ContainerDied","Data":"1f8c1f86d7cf663f296e209b57054cd1177014a3b2e96ee4a6b54c20234a4129"} Jul 21 20:22:10.756750 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.756730 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:22:10.789109 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.789086 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrf7n\" (UniqueName: \"kubernetes.io/projected/79a5cc8f-26cb-4ed9-b0d4-a11717742808-kube-api-access-vrf7n\") pod \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " Jul 21 20:22:10.789255 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.789153 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-util\") pod \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " Jul 21 20:22:10.789255 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.789214 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-bundle\") pod \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\" (UID: \"79a5cc8f-26cb-4ed9-b0d4-a11717742808\") " Jul 21 20:22:10.789568 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.789539 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-bundle" (OuterVolumeSpecName: "bundle") pod "79a5cc8f-26cb-4ed9-b0d4-a11717742808" (UID: "79a5cc8f-26cb-4ed9-b0d4-a11717742808"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:22:10.791140 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.791118 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a5cc8f-26cb-4ed9-b0d4-a11717742808-kube-api-access-vrf7n" (OuterVolumeSpecName: "kube-api-access-vrf7n") pod "79a5cc8f-26cb-4ed9-b0d4-a11717742808" (UID: "79a5cc8f-26cb-4ed9-b0d4-a11717742808"). InnerVolumeSpecName "kube-api-access-vrf7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:22:10.793745 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.793723 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-util" (OuterVolumeSpecName: "util") pod "79a5cc8f-26cb-4ed9-b0d4-a11717742808" (UID: "79a5cc8f-26cb-4ed9-b0d4-a11717742808"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:22:10.890261 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.890196 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrf7n\" (UniqueName: \"kubernetes.io/projected/79a5cc8f-26cb-4ed9-b0d4-a11717742808-kube-api-access-vrf7n\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:22:10.890261 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.890222 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-util\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:22:10.890261 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:10.890237 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79a5cc8f-26cb-4ed9-b0d4-a11717742808-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:22:11.637668 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:11.637638 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" Jul 21 20:22:11.637668 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:11.637658 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxj2w7" event={"ID":"79a5cc8f-26cb-4ed9-b0d4-a11717742808","Type":"ContainerDied","Data":"0b98154017f7b0a01ab901a27fca84a1a508733e516545a0849de20e3872b5d1"} Jul 21 20:22:11.637886 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:11.637704 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b98154017f7b0a01ab901a27fca84a1a508733e516545a0849de20e3872b5d1" Jul 21 20:22:16.383658 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.383624 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72"] Jul 21 20:22:16.384230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.384133 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerName="util" Jul 21 20:22:16.384230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.384153 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerName="util" Jul 21 20:22:16.384230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.384172 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerName="pull" Jul 21 20:22:16.384230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.384181 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerName="pull" Jul 21 20:22:16.384230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.384211 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerName="extract" Jul 21 20:22:16.384230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.384219 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerName="extract" Jul 21 20:22:16.384519 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.384300 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a5cc8f-26cb-4ed9-b0d4-a11717742808" containerName="extract" Jul 21 20:22:16.387417 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.387398 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" Jul 21 20:22:16.391509 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.391482 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 20:22:16.391612 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.391549 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 20:22:16.392245 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.392222 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hhg6l\"" Jul 21 20:22:16.399090 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.399067 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72"] Jul 21 20:22:16.441075 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.441043 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ptx7\" (UniqueName: \"kubernetes.io/projected/23ed2c97-e359-4f03-9c62-abaac45193d3-kube-api-access-8ptx7\") pod \"openshift-lws-operator-fd8ccff4c-h7v72\" (UID: \"23ed2c97-e359-4f03-9c62-abaac45193d3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" Jul 21 20:22:16.441223 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.441105 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23ed2c97-e359-4f03-9c62-abaac45193d3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-h7v72\" (UID: \"23ed2c97-e359-4f03-9c62-abaac45193d3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" Jul 21 20:22:16.542114 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.542082 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23ed2c97-e359-4f03-9c62-abaac45193d3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-h7v72\" (UID: \"23ed2c97-e359-4f03-9c62-abaac45193d3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" Jul 21 20:22:16.542265 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.542169 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ptx7\" (UniqueName: \"kubernetes.io/projected/23ed2c97-e359-4f03-9c62-abaac45193d3-kube-api-access-8ptx7\") pod \"openshift-lws-operator-fd8ccff4c-h7v72\" (UID: \"23ed2c97-e359-4f03-9c62-abaac45193d3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" Jul 21 20:22:16.542435 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.542417 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/23ed2c97-e359-4f03-9c62-abaac45193d3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-h7v72\" (UID: \"23ed2c97-e359-4f03-9c62-abaac45193d3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" Jul 21 20:22:16.551445 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.551415 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ptx7\" (UniqueName: \"kubernetes.io/projected/23ed2c97-e359-4f03-9c62-abaac45193d3-kube-api-access-8ptx7\") pod \"openshift-lws-operator-fd8ccff4c-h7v72\" (UID: \"23ed2c97-e359-4f03-9c62-abaac45193d3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" Jul 21 20:22:16.697174 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.697098 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" Jul 21 20:22:16.833138 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:16.833111 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72"] Jul 21 20:22:16.834674 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:22:16.834641 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23ed2c97_e359_4f03_9c62_abaac45193d3.slice/crio-4dc967bfd8f07930aa8fac19fd26a66e48c4c6f00d8591182d983960e3a3ee50 WatchSource:0}: Error finding container 4dc967bfd8f07930aa8fac19fd26a66e48c4c6f00d8591182d983960e3a3ee50: Status 404 returned error can't find the container with id 4dc967bfd8f07930aa8fac19fd26a66e48c4c6f00d8591182d983960e3a3ee50 Jul 21 20:22:17.667314 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:17.667275 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" event={"ID":"23ed2c97-e359-4f03-9c62-abaac45193d3","Type":"ContainerStarted","Data":"4dc967bfd8f07930aa8fac19fd26a66e48c4c6f00d8591182d983960e3a3ee50"} Jul 21 20:22:19.680768 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:19.680681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" event={"ID":"23ed2c97-e359-4f03-9c62-abaac45193d3","Type":"ContainerStarted","Data":"a4776cdcedce03fe3c8f24543540df1dd7453b27e3962b97a9fa574ba6880150"} Jul 21 20:22:19.699118 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:19.699061 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-h7v72" podStartSLOduration=1.119202565 podStartE2EDuration="3.699041868s" totalImagesPullingTime="2.579839303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:22:19.696996162 +0000 UTC m=+494.377122598" watchObservedRunningTime="2026-07-21 20:22:19.699041868 +0000 UTC m=+494.379168305" Jul 21 20:22:22.264607 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.264574 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z"] Jul 21 20:22:22.268252 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.268236 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.271124 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.271105 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 20:22:22.271250 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.271234 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 20:22:22.271901 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.271889 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fsfl2\"" Jul 21 20:22:22.281167 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.281143 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z"] Jul 21 20:22:22.391706 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.391666 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.391889 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.391727 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjv89\" (UniqueName: \"kubernetes.io/projected/6347fdd5-603e-467f-b9eb-bca819a0577d-kube-api-access-xjv89\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.391889 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.391789 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.492808 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.492773 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.492987 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.492846 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.492987 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.492882 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjv89\" (UniqueName: \"kubernetes.io/projected/6347fdd5-603e-467f-b9eb-bca819a0577d-kube-api-access-xjv89\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.493310 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.493287 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.493359 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.493303 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.501992 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.501968 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjv89\" (UniqueName: \"kubernetes.io/projected/6347fdd5-603e-467f-b9eb-bca819a0577d-kube-api-access-xjv89\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.577661 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.577582 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:22.728956 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:22.728913 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z"] Jul 21 20:22:22.732581 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:22:22.732552 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6347fdd5_603e_467f_b9eb_bca819a0577d.slice/crio-2b261d66dc7749b191d70ac431481120094770243dc58861ab6807b51d12b36b WatchSource:0}: Error finding container 2b261d66dc7749b191d70ac431481120094770243dc58861ab6807b51d12b36b: Status 404 returned error can't find the container with id 2b261d66dc7749b191d70ac431481120094770243dc58861ab6807b51d12b36b Jul 21 20:22:23.702510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:23.702469 2567 generic.go:356] "Generic (PLEG): container finished" podID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerID="436ac97d0b625134a058942ff0652dfbee0b676c8a18c0b779c695ec6974bd66" exitCode=0 Jul 21 20:22:23.702891 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:23.702549 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" event={"ID":"6347fdd5-603e-467f-b9eb-bca819a0577d","Type":"ContainerDied","Data":"436ac97d0b625134a058942ff0652dfbee0b676c8a18c0b779c695ec6974bd66"} Jul 21 20:22:23.702891 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:23.702628 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" event={"ID":"6347fdd5-603e-467f-b9eb-bca819a0577d","Type":"ContainerStarted","Data":"2b261d66dc7749b191d70ac431481120094770243dc58861ab6807b51d12b36b"} Jul 21 20:22:24.709807 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:24.709772 2567 generic.go:356] "Generic (PLEG): container finished" podID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerID="676a4e8a2a7dc6f49b1427aee45bf2d3a5dbb471a94f6b5f7ca7628f20696ac0" exitCode=0 Jul 21 20:22:24.710210 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:24.709863 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" event={"ID":"6347fdd5-603e-467f-b9eb-bca819a0577d","Type":"ContainerDied","Data":"676a4e8a2a7dc6f49b1427aee45bf2d3a5dbb471a94f6b5f7ca7628f20696ac0"} Jul 21 20:22:25.716245 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:25.716211 2567 generic.go:356] "Generic (PLEG): container finished" podID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerID="8fdb28ff44e156b04d2561e6d83159dc269b50132e794b7bc7fd228107ca6605" exitCode=0 Jul 21 20:22:25.716629 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:25.716299 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" event={"ID":"6347fdd5-603e-467f-b9eb-bca819a0577d","Type":"ContainerDied","Data":"8fdb28ff44e156b04d2561e6d83159dc269b50132e794b7bc7fd228107ca6605"} Jul 21 20:22:26.851645 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:26.851624 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:26.933360 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:26.933324 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-bundle\") pod \"6347fdd5-603e-467f-b9eb-bca819a0577d\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " Jul 21 20:22:26.933540 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:26.933393 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-util\") pod \"6347fdd5-603e-467f-b9eb-bca819a0577d\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " Jul 21 20:22:26.933540 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:26.933441 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjv89\" (UniqueName: \"kubernetes.io/projected/6347fdd5-603e-467f-b9eb-bca819a0577d-kube-api-access-xjv89\") pod \"6347fdd5-603e-467f-b9eb-bca819a0577d\" (UID: \"6347fdd5-603e-467f-b9eb-bca819a0577d\") " Jul 21 20:22:26.934079 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:26.934051 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-bundle" (OuterVolumeSpecName: "bundle") pod "6347fdd5-603e-467f-b9eb-bca819a0577d" (UID: "6347fdd5-603e-467f-b9eb-bca819a0577d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:22:26.935663 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:26.935638 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6347fdd5-603e-467f-b9eb-bca819a0577d-kube-api-access-xjv89" (OuterVolumeSpecName: "kube-api-access-xjv89") pod "6347fdd5-603e-467f-b9eb-bca819a0577d" (UID: "6347fdd5-603e-467f-b9eb-bca819a0577d"). InnerVolumeSpecName "kube-api-access-xjv89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:22:26.938743 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:26.938721 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-util" (OuterVolumeSpecName: "util") pod "6347fdd5-603e-467f-b9eb-bca819a0577d" (UID: "6347fdd5-603e-467f-b9eb-bca819a0577d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:22:27.034549 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:27.034451 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-util\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:22:27.034549 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:27.034498 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjv89\" (UniqueName: \"kubernetes.io/projected/6347fdd5-603e-467f-b9eb-bca819a0577d-kube-api-access-xjv89\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:22:27.034549 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:27.034509 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6347fdd5-603e-467f-b9eb-bca819a0577d-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:22:27.727013 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:27.726982 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" Jul 21 20:22:27.727182 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:27.726986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzvh7z" event={"ID":"6347fdd5-603e-467f-b9eb-bca819a0577d","Type":"ContainerDied","Data":"2b261d66dc7749b191d70ac431481120094770243dc58861ab6807b51d12b36b"} Jul 21 20:22:27.727182 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:27.727138 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b261d66dc7749b191d70ac431481120094770243dc58861ab6807b51d12b36b" Jul 21 20:22:35.170695 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.170655 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp"] Jul 21 20:22:35.171225 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.171205 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerName="util" Jul 21 20:22:35.171306 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.171229 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerName="util" Jul 21 20:22:35.171306 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.171246 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerName="extract" Jul 21 20:22:35.171306 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.171255 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerName="extract" Jul 21 20:22:35.171306 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.171278 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerName="pull" Jul 21 20:22:35.171306 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.171287 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerName="pull" Jul 21 20:22:35.171541 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.171385 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6347fdd5-603e-467f-b9eb-bca819a0577d" containerName="extract" Jul 21 20:22:35.174439 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.174419 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.178422 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.178402 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 20:22:35.178543 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.178479 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qw48m\"" Jul 21 20:22:35.178769 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.178755 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 20:22:35.178908 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.178894 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 20:22:35.187918 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.187887 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp"] Jul 21 20:22:35.307859 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.307824 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-manager-config\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.308065 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.307880 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-cert\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.308129 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.308100 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-metrics-cert\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.308184 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.308162 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkgl\" (UniqueName: \"kubernetes.io/projected/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-kube-api-access-jvkgl\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.409255 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.409222 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-metrics-cert\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.409255 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.409268 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkgl\" (UniqueName: \"kubernetes.io/projected/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-kube-api-access-jvkgl\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.409499 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.409291 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-manager-config\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.409499 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.409320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-cert\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.410039 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.410015 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-manager-config\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.411694 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.411664 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-metrics-cert\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.411782 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.411698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-cert\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.436879 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.436821 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkgl\" (UniqueName: \"kubernetes.io/projected/bdc0f0fe-adf9-418d-bfbb-2e1034263fa7-kube-api-access-jvkgl\") pod \"lws-controller-manager-697fc85fc4-7mtgp\" (UID: \"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7\") " pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.484810 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.484781 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:35.618858 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.618834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp"] Jul 21 20:22:35.620936 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:22:35.620912 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc0f0fe_adf9_418d_bfbb_2e1034263fa7.slice/crio-f9d300d19c7e8aa8ce79178a4c2a2abfe9b96e7fa75ecca18e14eb9da1624334 WatchSource:0}: Error finding container f9d300d19c7e8aa8ce79178a4c2a2abfe9b96e7fa75ecca18e14eb9da1624334: Status 404 returned error can't find the container with id f9d300d19c7e8aa8ce79178a4c2a2abfe9b96e7fa75ecca18e14eb9da1624334 Jul 21 20:22:35.766630 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:35.766543 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" event={"ID":"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7","Type":"ContainerStarted","Data":"f9d300d19c7e8aa8ce79178a4c2a2abfe9b96e7fa75ecca18e14eb9da1624334"} Jul 21 20:22:37.777513 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:37.777480 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" event={"ID":"bdc0f0fe-adf9-418d-bfbb-2e1034263fa7","Type":"ContainerStarted","Data":"1ba94c7580fd6348e9b64003bef9fe60e547603f62a369e349635e79f66309c3"} Jul 21 20:22:37.778011 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:37.777550 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:37.796852 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:37.796789 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" podStartSLOduration=1.436624814 podStartE2EDuration="2.796772125s" totalImagesPullingTime="1.360147311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:22:37.796082958 +0000 UTC m=+512.476209396" watchObservedRunningTime="2026-07-21 20:22:37.796772125 +0000 UTC m=+512.476898562" Jul 21 20:22:38.395391 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.395360 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb"] Jul 21 20:22:38.400495 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.400476 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.403585 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.403564 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 20:22:38.404087 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.404064 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 20:22:38.404197 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.404168 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 20:22:38.404580 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.404563 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 20:22:38.404670 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.404652 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-btdnv\"" Jul 21 20:22:38.423634 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.423603 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb"] Jul 21 20:22:38.536823 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.536785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfsk\" (UniqueName: \"kubernetes.io/projected/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-kube-api-access-4gfsk\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.537017 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.536838 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-tmp\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.537017 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.536916 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-manifests\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.537017 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.536962 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.537136 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.537029 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.637429 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.637397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-tmp\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.637672 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.637444 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-manifests\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.637672 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.637472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.637672 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.637526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.637672 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.637568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfsk\" (UniqueName: \"kubernetes.io/projected/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-kube-api-access-4gfsk\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.638005 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.637972 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-manifests\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.639852 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.639813 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-tmp\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.639984 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.639889 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.640227 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.640210 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.648318 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.648267 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfsk\" (UniqueName: \"kubernetes.io/projected/90c84641-8fdc-452c-93b8-d1fe8b7d2f3c-kube-api-access-4gfsk\") pod \"opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb\" (UID: \"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c\") " pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.711215 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.711186 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:38.908742 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:38.908700 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb"] Jul 21 20:22:38.910410 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:22:38.910384 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c84641_8fdc_452c_93b8_d1fe8b7d2f3c.slice/crio-75ddaa5a46150ef7a8143177a7e9a62c1fcd3fb944dcd17e44abcee191025001 WatchSource:0}: Error finding container 75ddaa5a46150ef7a8143177a7e9a62c1fcd3fb944dcd17e44abcee191025001: Status 404 returned error can't find the container with id 75ddaa5a46150ef7a8143177a7e9a62c1fcd3fb944dcd17e44abcee191025001 Jul 21 20:22:39.795040 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:39.794986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" event={"ID":"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c","Type":"ContainerStarted","Data":"75ddaa5a46150ef7a8143177a7e9a62c1fcd3fb944dcd17e44abcee191025001"} Jul 21 20:22:42.811308 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:42.811266 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" event={"ID":"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c","Type":"ContainerStarted","Data":"3aca0a820eae63f80d136c2215bb46d271820ae56b385811ad16387b38cdae53"} Jul 21 20:22:43.816263 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:43.816231 2567 generic.go:356] "Generic (PLEG): container finished" podID="90c84641-8fdc-452c-93b8-d1fe8b7d2f3c" containerID="3aca0a820eae63f80d136c2215bb46d271820ae56b385811ad16387b38cdae53" exitCode=0 Jul 21 20:22:43.816671 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:43.816314 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" event={"ID":"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c","Type":"ContainerDied","Data":"3aca0a820eae63f80d136c2215bb46d271820ae56b385811ad16387b38cdae53"} Jul 21 20:22:44.822981 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:44.822928 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" event={"ID":"90c84641-8fdc-452c-93b8-d1fe8b7d2f3c","Type":"ContainerStarted","Data":"b1368bae371a65217a9b25237052be8ef2afcd1238485a9c9532555043f67aaa"} Jul 21 20:22:44.823514 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:44.823029 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:44.849073 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:44.849020 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" podStartSLOduration=2.5134583619999997 podStartE2EDuration="6.849004804s" totalImagesPullingTime="3.272868819s" totalInitContainerRuntime="1.062677623s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:22:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:22:44.847139503 +0000 UTC m=+519.527265951" watchObservedRunningTime="2026-07-21 20:22:44.849004804 +0000 UTC m=+519.529131241" Jul 21 20:22:48.789765 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:48.789736 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-697fc85fc4-7mtgp" Jul 21 20:22:55.830324 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:55.830292 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb" Jul 21 20:22:58.300539 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.300465 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-567c747899-w859j"] Jul 21 20:22:58.309088 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.309065 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.313201 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.313178 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 20:22:58.313339 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.313201 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 20:22:58.314119 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.314102 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 20:22:58.314203 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.314185 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lfz6j\"" Jul 21 20:22:58.316384 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.316360 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 20:22:58.338658 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.338632 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-567c747899-w859j"] Jul 21 20:22:58.400274 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.400247 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52cb6c66-80e9-40cd-a37d-fd57cba871a0-tls-certs\") pod \"kube-auth-proxy-567c747899-w859j\" (UID: \"52cb6c66-80e9-40cd-a37d-fd57cba871a0\") " pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.400420 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.400294 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52cb6c66-80e9-40cd-a37d-fd57cba871a0-tmp\") pod \"kube-auth-proxy-567c747899-w859j\" (UID: \"52cb6c66-80e9-40cd-a37d-fd57cba871a0\") " pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.400420 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.400400 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ww9s\" (UniqueName: \"kubernetes.io/projected/52cb6c66-80e9-40cd-a37d-fd57cba871a0-kube-api-access-8ww9s\") pod \"kube-auth-proxy-567c747899-w859j\" (UID: \"52cb6c66-80e9-40cd-a37d-fd57cba871a0\") " pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.501569 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.501538 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52cb6c66-80e9-40cd-a37d-fd57cba871a0-tls-certs\") pod \"kube-auth-proxy-567c747899-w859j\" (UID: \"52cb6c66-80e9-40cd-a37d-fd57cba871a0\") " pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.501569 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.501571 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52cb6c66-80e9-40cd-a37d-fd57cba871a0-tmp\") pod \"kube-auth-proxy-567c747899-w859j\" (UID: \"52cb6c66-80e9-40cd-a37d-fd57cba871a0\") " pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.501790 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.501609 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ww9s\" (UniqueName: \"kubernetes.io/projected/52cb6c66-80e9-40cd-a37d-fd57cba871a0-kube-api-access-8ww9s\") pod \"kube-auth-proxy-567c747899-w859j\" (UID: \"52cb6c66-80e9-40cd-a37d-fd57cba871a0\") " pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.503870 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.503847 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52cb6c66-80e9-40cd-a37d-fd57cba871a0-tmp\") pod \"kube-auth-proxy-567c747899-w859j\" (UID: \"52cb6c66-80e9-40cd-a37d-fd57cba871a0\") " pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.504005 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.503987 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52cb6c66-80e9-40cd-a37d-fd57cba871a0-tls-certs\") pod \"kube-auth-proxy-567c747899-w859j\" (UID: \"52cb6c66-80e9-40cd-a37d-fd57cba871a0\") " pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.528168 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.528144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ww9s\" (UniqueName: \"kubernetes.io/projected/52cb6c66-80e9-40cd-a37d-fd57cba871a0-kube-api-access-8ww9s\") pod \"kube-auth-proxy-567c747899-w859j\" (UID: \"52cb6c66-80e9-40cd-a37d-fd57cba871a0\") " pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.619352 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.619330 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" Jul 21 20:22:58.828708 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:22:58.828679 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52cb6c66_80e9_40cd_a37d_fd57cba871a0.slice/crio-fd96f542d5c0aea2fcbea4cbbf19f2a682cb184c1219b31c70d2b9739d143961 WatchSource:0}: Error finding container fd96f542d5c0aea2fcbea4cbbf19f2a682cb184c1219b31c70d2b9739d143961: Status 404 returned error can't find the container with id fd96f542d5c0aea2fcbea4cbbf19f2a682cb184c1219b31c70d2b9739d143961 Jul 21 20:22:58.834322 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.834299 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-567c747899-w859j"] Jul 21 20:22:58.894850 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:22:58.894769 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" event={"ID":"52cb6c66-80e9-40cd-a37d-fd57cba871a0","Type":"ContainerStarted","Data":"fd96f542d5c0aea2fcbea4cbbf19f2a682cb184c1219b31c70d2b9739d143961"} Jul 21 20:23:01.919452 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:23:01.919416 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" event={"ID":"52cb6c66-80e9-40cd-a37d-fd57cba871a0","Type":"ContainerStarted","Data":"a86456db66ebb131565f9f4115b6869c571b7b13849d08a8844c6bb11c7ad453"} Jul 21 20:23:01.938179 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:23:01.938127 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-567c747899-w859j" podStartSLOduration=1.035494244 podStartE2EDuration="3.938112592s" totalImagesPullingTime="2.902618348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:22:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:23:01.936321174 +0000 UTC m=+536.616447625" watchObservedRunningTime="2026-07-21 20:23:01.938112592 +0000 UTC m=+536.618239028" Jul 21 20:24:05.714900 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:05.714872 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:24:05.716325 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:05.716306 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:24:13.609458 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.609426 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-846bf86f88-8g5vp"] Jul 21 20:24:13.612646 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.612630 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.626882 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.626858 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846bf86f88-8g5vp"] Jul 21 20:24:13.779286 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.779258 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-trusted-ca-bundle\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.779286 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.779290 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-oauth-serving-cert\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.779516 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.779306 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-console-config\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.779516 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.779327 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61e462c0-8f16-405e-8f0c-ae8f9929e315-console-serving-cert\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.779516 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.779426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61e462c0-8f16-405e-8f0c-ae8f9929e315-console-oauth-config\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.779516 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.779487 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-service-ca\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.779656 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.779531 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjv28\" (UniqueName: \"kubernetes.io/projected/61e462c0-8f16-405e-8f0c-ae8f9929e315-kube-api-access-jjv28\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.880325 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.880296 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjv28\" (UniqueName: \"kubernetes.io/projected/61e462c0-8f16-405e-8f0c-ae8f9929e315-kube-api-access-jjv28\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.880510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.880333 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-trusted-ca-bundle\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.880510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.880364 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-oauth-serving-cert\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.880510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.880392 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-console-config\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.880510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.880421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61e462c0-8f16-405e-8f0c-ae8f9929e315-console-serving-cert\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.880510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.880446 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61e462c0-8f16-405e-8f0c-ae8f9929e315-console-oauth-config\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.880510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.880500 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-service-ca\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.881482 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.881458 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-oauth-serving-cert\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.881591 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.881544 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-service-ca\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.881591 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.881555 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-console-config\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.881667 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.881626 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61e462c0-8f16-405e-8f0c-ae8f9929e315-trusted-ca-bundle\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.883100 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.883078 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61e462c0-8f16-405e-8f0c-ae8f9929e315-console-oauth-config\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.883252 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.883235 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61e462c0-8f16-405e-8f0c-ae8f9929e315-console-serving-cert\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.891070 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.891052 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjv28\" (UniqueName: \"kubernetes.io/projected/61e462c0-8f16-405e-8f0c-ae8f9929e315-kube-api-access-jjv28\") pod \"console-846bf86f88-8g5vp\" (UID: \"61e462c0-8f16-405e-8f0c-ae8f9929e315\") " pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:13.922068 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:13.922047 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:14.057757 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:14.057732 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846bf86f88-8g5vp"] Jul 21 20:24:14.060264 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:24:14.060233 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e462c0_8f16_405e_8f0c_ae8f9929e315.slice/crio-0fa5f97619b4d6637f02eff2c6dbb569a68a85c1e6cb15e3978d5c405c1bd147 WatchSource:0}: Error finding container 0fa5f97619b4d6637f02eff2c6dbb569a68a85c1e6cb15e3978d5c405c1bd147: Status 404 returned error can't find the container with id 0fa5f97619b4d6637f02eff2c6dbb569a68a85c1e6cb15e3978d5c405c1bd147 Jul 21 20:24:14.278928 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:14.278834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846bf86f88-8g5vp" event={"ID":"61e462c0-8f16-405e-8f0c-ae8f9929e315","Type":"ContainerStarted","Data":"71edf7fc44177b12976d71dfce20dd1440892031045e5b6e552f8643ad5f89fa"} Jul 21 20:24:14.278928 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:14.278893 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846bf86f88-8g5vp" event={"ID":"61e462c0-8f16-405e-8f0c-ae8f9929e315","Type":"ContainerStarted","Data":"0fa5f97619b4d6637f02eff2c6dbb569a68a85c1e6cb15e3978d5c405c1bd147"} Jul 21 20:24:14.303075 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:14.303022 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-846bf86f88-8g5vp" podStartSLOduration=1.303007695 podStartE2EDuration="1.303007695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:24:14.301188321 +0000 UTC m=+608.981314781" watchObservedRunningTime="2026-07-21 20:24:14.303007695 +0000 UTC m=+608.983134133" Jul 21 20:24:17.011713 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.011676 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7"] Jul 21 20:24:17.015398 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.015381 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.018910 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.018892 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 20:24:17.019806 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.019757 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-phzwh\"" Jul 21 20:24:17.019806 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.019775 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 20:24:17.026629 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.026607 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7"] Jul 21 20:24:17.110379 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.110345 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.110379 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.110382 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.110586 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.110410 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2g46\" (UniqueName: \"kubernetes.io/projected/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-kube-api-access-j2g46\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.211008 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.210973 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.211162 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.211017 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.211162 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.211045 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2g46\" (UniqueName: \"kubernetes.io/projected/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-kube-api-access-j2g46\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.211379 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.211362 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.211428 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.211409 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.235072 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.235039 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2g46\" (UniqueName: \"kubernetes.io/projected/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-kube-api-access-j2g46\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.325859 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.325770 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:17.334858 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.334829 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8"] Jul 21 20:24:17.338548 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.338532 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.355699 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.355669 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8"] Jul 21 20:24:17.472200 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.472169 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7"] Jul 21 20:24:17.474406 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:24:17.474370 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60a7b4d_aa89_48e5_bc0c_6dfcca412956.slice/crio-5a4aa0761b3273a108e86ad1b20a571a9c2b329d31bb957c50a0712f821b70c9 WatchSource:0}: Error finding container 5a4aa0761b3273a108e86ad1b20a571a9c2b329d31bb957c50a0712f821b70c9: Status 404 returned error can't find the container with id 5a4aa0761b3273a108e86ad1b20a571a9c2b329d31bb957c50a0712f821b70c9 Jul 21 20:24:17.513111 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.513083 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.513244 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.513203 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4xv\" (UniqueName: \"kubernetes.io/projected/12c01489-998c-4a0f-b70e-a80c9e05c43f-kube-api-access-fq4xv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.513292 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.513261 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.614085 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.614050 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.614246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.614103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.614246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.614168 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4xv\" (UniqueName: \"kubernetes.io/projected/12c01489-998c-4a0f-b70e-a80c9e05c43f-kube-api-access-fq4xv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.614453 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.614433 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.614514 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.614498 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.626153 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.626121 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4xv\" (UniqueName: \"kubernetes.io/projected/12c01489-998c-4a0f-b70e-a80c9e05c43f-kube-api-access-fq4xv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.660597 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.660563 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:17.764853 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.764817 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k"] Jul 21 20:24:17.768976 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.768931 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:17.784092 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.784062 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k"] Jul 21 20:24:17.822348 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.822322 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8"] Jul 21 20:24:17.917366 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.917323 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:17.917537 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.917417 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjv6v\" (UniqueName: \"kubernetes.io/projected/966e8087-e229-4c1e-8011-c61e3e8b2b15-kube-api-access-hjv6v\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:17.917537 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:17.917453 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:18.018584 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.018546 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:18.019036 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.018693 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjv6v\" (UniqueName: \"kubernetes.io/projected/966e8087-e229-4c1e-8011-c61e3e8b2b15-kube-api-access-hjv6v\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:18.019036 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.018729 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:18.019036 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.018974 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:18.019159 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.019119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:18.033864 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.033813 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjv6v\" (UniqueName: \"kubernetes.io/projected/966e8087-e229-4c1e-8011-c61e3e8b2b15-kube-api-access-hjv6v\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:18.079285 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.079199 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:18.107272 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.107245 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf"] Jul 21 20:24:18.111073 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.111050 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.148857 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.148825 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf"] Jul 21 20:24:18.220673 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.220641 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.220836 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.220709 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjg8g\" (UniqueName: \"kubernetes.io/projected/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-kube-api-access-hjg8g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.220836 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.220737 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.224368 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.224291 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k"] Jul 21 20:24:18.226586 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:24:18.226564 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966e8087_e229_4c1e_8011_c61e3e8b2b15.slice/crio-9a32b97b3016ed20d6c7dc930e9e835bfc49c8b4d8319b726634b0f8217347a7 WatchSource:0}: Error finding container 9a32b97b3016ed20d6c7dc930e9e835bfc49c8b4d8319b726634b0f8217347a7: Status 404 returned error can't find the container with id 9a32b97b3016ed20d6c7dc930e9e835bfc49c8b4d8319b726634b0f8217347a7 Jul 21 20:24:18.299882 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.299850 2567 generic.go:356] "Generic (PLEG): container finished" podID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerID="d83be67785dde9deb84452a6243355148c528829f8acbeace57311798dea13bd" exitCode=0 Jul 21 20:24:18.300041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.299934 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" event={"ID":"12c01489-998c-4a0f-b70e-a80c9e05c43f","Type":"ContainerDied","Data":"d83be67785dde9deb84452a6243355148c528829f8acbeace57311798dea13bd"} Jul 21 20:24:18.300041 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.300015 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" event={"ID":"12c01489-998c-4a0f-b70e-a80c9e05c43f","Type":"ContainerStarted","Data":"2f89f3a5a01878d52134d6f116c9babb29d59a3b54864365491492fb6aed5c29"} Jul 21 20:24:18.300460 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.300444 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 20:24:18.301648 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.301629 2567 generic.go:356] "Generic (PLEG): container finished" podID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerID="939296ec2015c62a2b393587b49a3299a857a4aeb28563de6272eb84fa783153" exitCode=0 Jul 21 20:24:18.301764 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.301709 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" event={"ID":"a60a7b4d-aa89-48e5-bc0c-6dfcca412956","Type":"ContainerDied","Data":"939296ec2015c62a2b393587b49a3299a857a4aeb28563de6272eb84fa783153"} Jul 21 20:24:18.301764 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.301753 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" event={"ID":"a60a7b4d-aa89-48e5-bc0c-6dfcca412956","Type":"ContainerStarted","Data":"5a4aa0761b3273a108e86ad1b20a571a9c2b329d31bb957c50a0712f821b70c9"} Jul 21 20:24:18.308236 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.308213 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" event={"ID":"966e8087-e229-4c1e-8011-c61e3e8b2b15","Type":"ContainerStarted","Data":"9a32b97b3016ed20d6c7dc930e9e835bfc49c8b4d8319b726634b0f8217347a7"} Jul 21 20:24:18.321311 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.321290 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjg8g\" (UniqueName: \"kubernetes.io/projected/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-kube-api-access-hjg8g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.321410 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.321320 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.321465 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.321409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.322150 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.321795 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.322150 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.321853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.330479 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.330421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjg8g\" (UniqueName: \"kubernetes.io/projected/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-kube-api-access-hjg8g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.420202 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.420168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:18.579496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:18.579467 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf"] Jul 21 20:24:18.580421 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:24:18.580392 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e0ddd3_0a2c_4ec8_ac0d_fb3faeebb66b.slice/crio-9b082e7404a863c00cad33a8729661f4d1593da33fe098912e750cd609aa1781 WatchSource:0}: Error finding container 9b082e7404a863c00cad33a8729661f4d1593da33fe098912e750cd609aa1781: Status 404 returned error can't find the container with id 9b082e7404a863c00cad33a8729661f4d1593da33fe098912e750cd609aa1781 Jul 21 20:24:19.317459 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:19.317429 2567 generic.go:356] "Generic (PLEG): container finished" podID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerID="8564c537f8c281a5856ee2275f4f16d9bc1938d92a2dee7989794d58ba2f5563" exitCode=0 Jul 21 20:24:19.317875 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:19.317512 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" event={"ID":"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b","Type":"ContainerDied","Data":"8564c537f8c281a5856ee2275f4f16d9bc1938d92a2dee7989794d58ba2f5563"} Jul 21 20:24:19.317875 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:19.317560 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" event={"ID":"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b","Type":"ContainerStarted","Data":"9b082e7404a863c00cad33a8729661f4d1593da33fe098912e750cd609aa1781"} Jul 21 20:24:19.319509 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:19.319488 2567 generic.go:356] "Generic (PLEG): container finished" podID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerID="8c168e04576aa1f199f67adf255f1420e6d26be8f617d47d32317fe14492d9be" exitCode=0 Jul 21 20:24:19.319617 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:19.319574 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" event={"ID":"12c01489-998c-4a0f-b70e-a80c9e05c43f","Type":"ContainerDied","Data":"8c168e04576aa1f199f67adf255f1420e6d26be8f617d47d32317fe14492d9be"} Jul 21 20:24:19.321510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:19.321487 2567 generic.go:356] "Generic (PLEG): container finished" podID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerID="91e701bcd45203257fc72bca3e1949660849b27fa8bd444f78a561c5925152bd" exitCode=0 Jul 21 20:24:19.321608 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:19.321518 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" event={"ID":"a60a7b4d-aa89-48e5-bc0c-6dfcca412956","Type":"ContainerDied","Data":"91e701bcd45203257fc72bca3e1949660849b27fa8bd444f78a561c5925152bd"} Jul 21 20:24:19.323220 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:19.323200 2567 generic.go:356] "Generic (PLEG): container finished" podID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerID="c2b9e8c9a0083521c7bf54e439db8d697d06e8fbdb10005c52c9f3ced1905c89" exitCode=0 Jul 21 20:24:19.323288 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:19.323249 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" event={"ID":"966e8087-e229-4c1e-8011-c61e3e8b2b15","Type":"ContainerDied","Data":"c2b9e8c9a0083521c7bf54e439db8d697d06e8fbdb10005c52c9f3ced1905c89"} Jul 21 20:24:20.336198 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:20.336165 2567 generic.go:356] "Generic (PLEG): container finished" podID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerID="63d3f1c50ca086a674204b7df337f19647517ef6c97e920bc64bf23aac30b4ba" exitCode=0 Jul 21 20:24:20.336610 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:20.336241 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" event={"ID":"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b","Type":"ContainerDied","Data":"63d3f1c50ca086a674204b7df337f19647517ef6c97e920bc64bf23aac30b4ba"} Jul 21 20:24:20.338458 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:20.338417 2567 generic.go:356] "Generic (PLEG): container finished" podID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerID="925e8a4e622cef9005393c6c4f51d1aef3f4e53fed599f9265165bbf27b5a76e" exitCode=0 Jul 21 20:24:20.338554 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:20.338460 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" event={"ID":"12c01489-998c-4a0f-b70e-a80c9e05c43f","Type":"ContainerDied","Data":"925e8a4e622cef9005393c6c4f51d1aef3f4e53fed599f9265165bbf27b5a76e"} Jul 21 20:24:20.340576 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:20.340554 2567 generic.go:356] "Generic (PLEG): container finished" podID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerID="490a9bb5a82e8768f68f5129809599ce9237b61069d7e3cbcd965db1293da0ed" exitCode=0 Jul 21 20:24:20.340643 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:20.340584 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" event={"ID":"a60a7b4d-aa89-48e5-bc0c-6dfcca412956","Type":"ContainerDied","Data":"490a9bb5a82e8768f68f5129809599ce9237b61069d7e3cbcd965db1293da0ed"} Jul 21 20:24:20.342428 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:20.342410 2567 generic.go:356] "Generic (PLEG): container finished" podID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerID="35a9b063beef056cceb70001d965bd047dc400c06764390d79d131f52ffb6bfa" exitCode=0 Jul 21 20:24:20.342502 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:20.342479 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" event={"ID":"966e8087-e229-4c1e-8011-c61e3e8b2b15","Type":"ContainerDied","Data":"35a9b063beef056cceb70001d965bd047dc400c06764390d79d131f52ffb6bfa"} Jul 21 20:24:21.350700 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.350663 2567 generic.go:356] "Generic (PLEG): container finished" podID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerID="e241350ddccaf13a75f665f8fedf3a480f14c5c253a6aa5788389115ed095ee0" exitCode=0 Jul 21 20:24:21.351173 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.350741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" event={"ID":"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b","Type":"ContainerDied","Data":"e241350ddccaf13a75f665f8fedf3a480f14c5c253a6aa5788389115ed095ee0"} Jul 21 20:24:21.352840 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.352819 2567 generic.go:356] "Generic (PLEG): container finished" podID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerID="8f826728dace57550f95a53b48769cd63b62c1513ce547059832b88449cb562d" exitCode=0 Jul 21 20:24:21.352982 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.352846 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" event={"ID":"966e8087-e229-4c1e-8011-c61e3e8b2b15","Type":"ContainerDied","Data":"8f826728dace57550f95a53b48769cd63b62c1513ce547059832b88449cb562d"} Jul 21 20:24:21.509912 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.509893 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:21.512707 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.512688 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:21.651501 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.651475 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-bundle\") pod \"12c01489-998c-4a0f-b70e-a80c9e05c43f\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " Jul 21 20:24:21.651644 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.651529 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-util\") pod \"12c01489-998c-4a0f-b70e-a80c9e05c43f\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " Jul 21 20:24:21.651644 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.651557 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2g46\" (UniqueName: \"kubernetes.io/projected/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-kube-api-access-j2g46\") pod \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " Jul 21 20:24:21.651644 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.651588 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq4xv\" (UniqueName: \"kubernetes.io/projected/12c01489-998c-4a0f-b70e-a80c9e05c43f-kube-api-access-fq4xv\") pod \"12c01489-998c-4a0f-b70e-a80c9e05c43f\" (UID: \"12c01489-998c-4a0f-b70e-a80c9e05c43f\") " Jul 21 20:24:21.651795 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.651680 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-util\") pod \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " Jul 21 20:24:21.651795 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.651714 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-bundle\") pod \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\" (UID: \"a60a7b4d-aa89-48e5-bc0c-6dfcca412956\") " Jul 21 20:24:21.652254 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.652226 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-bundle" (OuterVolumeSpecName: "bundle") pod "a60a7b4d-aa89-48e5-bc0c-6dfcca412956" (UID: "a60a7b4d-aa89-48e5-bc0c-6dfcca412956"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:24:21.652254 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.652245 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-bundle" (OuterVolumeSpecName: "bundle") pod "12c01489-998c-4a0f-b70e-a80c9e05c43f" (UID: "12c01489-998c-4a0f-b70e-a80c9e05c43f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:24:21.654406 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.654381 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-kube-api-access-j2g46" (OuterVolumeSpecName: "kube-api-access-j2g46") pod "a60a7b4d-aa89-48e5-bc0c-6dfcca412956" (UID: "a60a7b4d-aa89-48e5-bc0c-6dfcca412956"). InnerVolumeSpecName "kube-api-access-j2g46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:24:21.654496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.654401 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c01489-998c-4a0f-b70e-a80c9e05c43f-kube-api-access-fq4xv" (OuterVolumeSpecName: "kube-api-access-fq4xv") pod "12c01489-998c-4a0f-b70e-a80c9e05c43f" (UID: "12c01489-998c-4a0f-b70e-a80c9e05c43f"). InnerVolumeSpecName "kube-api-access-fq4xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:24:21.657354 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.657334 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-util" (OuterVolumeSpecName: "util") pod "12c01489-998c-4a0f-b70e-a80c9e05c43f" (UID: "12c01489-998c-4a0f-b70e-a80c9e05c43f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:24:21.658262 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.658238 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-util" (OuterVolumeSpecName: "util") pod "a60a7b4d-aa89-48e5-bc0c-6dfcca412956" (UID: "a60a7b4d-aa89-48e5-bc0c-6dfcca412956"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:24:21.752785 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.752705 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-util\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:21.752785 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.752732 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2g46\" (UniqueName: \"kubernetes.io/projected/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-kube-api-access-j2g46\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:21.752785 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.752745 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4xv\" (UniqueName: \"kubernetes.io/projected/12c01489-998c-4a0f-b70e-a80c9e05c43f-kube-api-access-fq4xv\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:21.752785 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.752754 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-util\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:21.752785 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.752762 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a60a7b4d-aa89-48e5-bc0c-6dfcca412956-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:21.752785 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:21.752769 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12c01489-998c-4a0f-b70e-a80c9e05c43f-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:22.360636 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.360600 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" event={"ID":"12c01489-998c-4a0f-b70e-a80c9e05c43f","Type":"ContainerDied","Data":"2f89f3a5a01878d52134d6f116c9babb29d59a3b54864365491492fb6aed5c29"} Jul 21 20:24:22.361065 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.360651 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f89f3a5a01878d52134d6f116c9babb29d59a3b54864365491492fb6aed5c29" Jul 21 20:24:22.361065 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.360613 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8" Jul 21 20:24:22.362764 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.362734 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" Jul 21 20:24:22.362969 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.362777 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7" event={"ID":"a60a7b4d-aa89-48e5-bc0c-6dfcca412956","Type":"ContainerDied","Data":"5a4aa0761b3273a108e86ad1b20a571a9c2b329d31bb957c50a0712f821b70c9"} Jul 21 20:24:22.362969 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.362848 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a4aa0761b3273a108e86ad1b20a571a9c2b329d31bb957c50a0712f821b70c9" Jul 21 20:24:22.529649 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.529623 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:22.555524 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.555500 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:22.661524 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.661486 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-bundle\") pod \"966e8087-e229-4c1e-8011-c61e3e8b2b15\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " Jul 21 20:24:22.661524 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.661530 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjv6v\" (UniqueName: \"kubernetes.io/projected/966e8087-e229-4c1e-8011-c61e3e8b2b15-kube-api-access-hjv6v\") pod \"966e8087-e229-4c1e-8011-c61e3e8b2b15\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " Jul 21 20:24:22.661772 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.661555 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-util\") pod \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " Jul 21 20:24:22.661772 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.661592 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-bundle\") pod \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " Jul 21 20:24:22.661772 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.661629 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-util\") pod \"966e8087-e229-4c1e-8011-c61e3e8b2b15\" (UID: \"966e8087-e229-4c1e-8011-c61e3e8b2b15\") " Jul 21 20:24:22.661772 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.661671 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjg8g\" (UniqueName: \"kubernetes.io/projected/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-kube-api-access-hjg8g\") pod \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\" (UID: \"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b\") " Jul 21 20:24:22.662222 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.662188 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-bundle" (OuterVolumeSpecName: "bundle") pod "966e8087-e229-4c1e-8011-c61e3e8b2b15" (UID: "966e8087-e229-4c1e-8011-c61e3e8b2b15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:24:22.662327 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.662279 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-bundle" (OuterVolumeSpecName: "bundle") pod "d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" (UID: "d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:24:22.663818 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.663787 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966e8087-e229-4c1e-8011-c61e3e8b2b15-kube-api-access-hjv6v" (OuterVolumeSpecName: "kube-api-access-hjv6v") pod "966e8087-e229-4c1e-8011-c61e3e8b2b15" (UID: "966e8087-e229-4c1e-8011-c61e3e8b2b15"). InnerVolumeSpecName "kube-api-access-hjv6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:24:22.663893 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.663822 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-kube-api-access-hjg8g" (OuterVolumeSpecName: "kube-api-access-hjg8g") pod "d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" (UID: "d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b"). InnerVolumeSpecName "kube-api-access-hjg8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:24:22.667302 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.667266 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-util" (OuterVolumeSpecName: "util") pod "d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" (UID: "d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:24:22.669906 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.669881 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-util" (OuterVolumeSpecName: "util") pod "966e8087-e229-4c1e-8011-c61e3e8b2b15" (UID: "966e8087-e229-4c1e-8011-c61e3e8b2b15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 20:24:22.762418 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.762385 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:22.762418 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.762411 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjv6v\" (UniqueName: \"kubernetes.io/projected/966e8087-e229-4c1e-8011-c61e3e8b2b15-kube-api-access-hjv6v\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:22.762418 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.762422 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-util\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:22.762418 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.762431 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:22.762670 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.762439 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/966e8087-e229-4c1e-8011-c61e3e8b2b15-util\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:22.762670 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:22.762447 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjg8g\" (UniqueName: \"kubernetes.io/projected/d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b-kube-api-access-hjg8g\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:23.369925 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:23.369890 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" Jul 21 20:24:23.369925 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:23.369906 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k" event={"ID":"966e8087-e229-4c1e-8011-c61e3e8b2b15","Type":"ContainerDied","Data":"9a32b97b3016ed20d6c7dc930e9e835bfc49c8b4d8319b726634b0f8217347a7"} Jul 21 20:24:23.370452 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:23.369998 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a32b97b3016ed20d6c7dc930e9e835bfc49c8b4d8319b726634b0f8217347a7" Jul 21 20:24:23.371852 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:23.371830 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" event={"ID":"d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b","Type":"ContainerDied","Data":"9b082e7404a863c00cad33a8729661f4d1593da33fe098912e750cd609aa1781"} Jul 21 20:24:23.371852 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:23.371852 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf" Jul 21 20:24:23.371993 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:23.371866 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b082e7404a863c00cad33a8729661f4d1593da33fe098912e750cd609aa1781" Jul 21 20:24:23.923135 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:23.923093 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:23.923348 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:23.923155 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:23.927977 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:23.927929 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:24.381290 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:24.381258 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-846bf86f88-8g5vp" Jul 21 20:24:24.436765 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:24.436723 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679fd5ff7b-fkhgj"] Jul 21 20:24:49.463038 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.462909 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-679fd5ff7b-fkhgj" podUID="4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" containerName="console" containerID="cri-o://e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4" gracePeriod=15 Jul 21 20:24:49.719838 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.719780 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:24:49.820388 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820358 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-serving-cert\") pod \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " Jul 21 20:24:49.820553 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820395 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-config\") pod \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " Jul 21 20:24:49.820553 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820429 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-service-ca\") pod \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " Jul 21 20:24:49.820553 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820455 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-trusted-ca-bundle\") pod \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " Jul 21 20:24:49.820553 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820504 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-oauth-serving-cert\") pod \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " Jul 21 20:24:49.820553 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820542 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj89r\" (UniqueName: \"kubernetes.io/projected/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-kube-api-access-vj89r\") pod \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " Jul 21 20:24:49.820814 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820570 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-oauth-config\") pod \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\" (UID: \"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2\") " Jul 21 20:24:49.820874 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820840 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-config" (OuterVolumeSpecName: "console-config") pod "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" (UID: "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:24:49.820874 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820835 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" (UID: "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:24:49.821017 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.820978 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" (UID: "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:24:49.821169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.821138 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" (UID: "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 20:24:49.822682 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.822657 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" (UID: "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:24:49.822794 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.822708 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" (UID: "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 20:24:49.822794 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.822724 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-kube-api-access-vj89r" (OuterVolumeSpecName: "kube-api-access-vj89r") pod "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" (UID: "4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2"). InnerVolumeSpecName "kube-api-access-vj89r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:24:49.921459 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.921413 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:49.921459 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.921453 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:49.921459 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.921461 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-service-ca\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:49.921459 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.921469 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-trusted-ca-bundle\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:49.921459 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.921478 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-oauth-serving-cert\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:49.921769 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.921486 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj89r\" (UniqueName: \"kubernetes.io/projected/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-kube-api-access-vj89r\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:49.921769 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:49.921494 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2-console-oauth-config\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:24:50.525149 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:50.525112 2567 generic.go:356] "Generic (PLEG): container finished" podID="4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" containerID="e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4" exitCode=0 Jul 21 20:24:50.525585 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:50.525161 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-679fd5ff7b-fkhgj" Jul 21 20:24:50.525585 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:50.525200 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679fd5ff7b-fkhgj" event={"ID":"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2","Type":"ContainerDied","Data":"e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4"} Jul 21 20:24:50.525585 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:50.525254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-679fd5ff7b-fkhgj" event={"ID":"4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2","Type":"ContainerDied","Data":"1aefd698d0ff08f53f99efcbe8358ed498fb31338ac6730620569e33064f59cb"} Jul 21 20:24:50.525585 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:50.525277 2567 scope.go:117] "RemoveContainer" containerID="e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4" Jul 21 20:24:50.534767 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:50.534748 2567 scope.go:117] "RemoveContainer" containerID="e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4" Jul 21 20:24:50.535111 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:24:50.535093 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4\": container with ID starting with e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4 not found: ID does not exist" containerID="e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4" Jul 21 20:24:50.535178 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:50.535121 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4"} err="failed to get container status \"e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4\": rpc error: code = NotFound desc = could not find container \"e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4\": container with ID starting with e3649e49387cabcdbca004f8e658a12fcbe7e9a194bd9c32745410acebdf3bb4 not found: ID does not exist" Jul 21 20:24:50.556840 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:50.556781 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-679fd5ff7b-fkhgj"] Jul 21 20:24:50.559091 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:50.559067 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-679fd5ff7b-fkhgj"] Jul 21 20:24:51.805804 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:24:51.805757 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" path="/var/lib/kubelet/pods/4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2/volumes" Jul 21 20:25:42.107159 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107111 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:25:42.107706 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107627 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerName="util" Jul 21 20:25:42.107706 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107648 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerName="util" Jul 21 20:25:42.107706 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107671 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerName="util" Jul 21 20:25:42.107706 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107679 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerName="util" Jul 21 20:25:42.107706 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107692 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerName="pull" Jul 21 20:25:42.107706 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107699 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerName="pull" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107712 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" containerName="console" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107720 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" containerName="console" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107733 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerName="pull" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107743 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerName="pull" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107752 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107761 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107773 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerName="util" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107782 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerName="util" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107791 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107799 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107819 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerName="pull" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107827 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerName="pull" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107835 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107843 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107852 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107860 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107870 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerName="util" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107878 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerName="util" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107895 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerName="pull" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.107904 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerName="pull" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.108017 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a3002ab-abb1-4d70-ab45-dcd1bd4d74a2" containerName="console" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.108032 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60a7b4d-aa89-48e5-bc0c-6dfcca412956" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.108041 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c01489-998c-4a0f-b70e-a80c9e05c43f" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.108048 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b" containerName="extract" Jul 21 20:25:42.108077 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.108059 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="966e8087-e229-4c1e-8011-c61e3e8b2b15" containerName="extract" Jul 21 20:25:42.111576 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.111556 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:25:42.116457 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.116432 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-phzwh\"" Jul 21 20:25:42.116593 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.116494 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 20:25:42.117357 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.117330 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 20:25:42.117471 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.117332 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 20:25:42.126407 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.126384 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:25:42.192646 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.192604 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/62c1acb0-bd2e-49f5-866a-29fbb63ff00c-config-file\") pod \"limitador-limitador-54d6d988d5-bh898\" (UID: \"62c1acb0-bd2e-49f5-866a-29fbb63ff00c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:25:42.192846 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.192680 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ttx4\" (UniqueName: \"kubernetes.io/projected/62c1acb0-bd2e-49f5-866a-29fbb63ff00c-kube-api-access-7ttx4\") pod \"limitador-limitador-54d6d988d5-bh898\" (UID: \"62c1acb0-bd2e-49f5-866a-29fbb63ff00c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:25:42.203268 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.203232 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:25:42.293586 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.293546 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/62c1acb0-bd2e-49f5-866a-29fbb63ff00c-config-file\") pod \"limitador-limitador-54d6d988d5-bh898\" (UID: \"62c1acb0-bd2e-49f5-866a-29fbb63ff00c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:25:42.293769 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.293605 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ttx4\" (UniqueName: \"kubernetes.io/projected/62c1acb0-bd2e-49f5-866a-29fbb63ff00c-kube-api-access-7ttx4\") pod \"limitador-limitador-54d6d988d5-bh898\" (UID: \"62c1acb0-bd2e-49f5-866a-29fbb63ff00c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:25:42.294245 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.294225 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/62c1acb0-bd2e-49f5-866a-29fbb63ff00c-config-file\") pod \"limitador-limitador-54d6d988d5-bh898\" (UID: \"62c1acb0-bd2e-49f5-866a-29fbb63ff00c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:25:42.303547 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.303516 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ttx4\" (UniqueName: \"kubernetes.io/projected/62c1acb0-bd2e-49f5-866a-29fbb63ff00c-kube-api-access-7ttx4\") pod \"limitador-limitador-54d6d988d5-bh898\" (UID: \"62c1acb0-bd2e-49f5-866a-29fbb63ff00c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:25:42.422032 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.421980 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:25:42.592139 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.592110 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:25:42.593936 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:25:42.593910 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c1acb0_bd2e_49f5_866a_29fbb63ff00c.slice/crio-aafccbecfe41446a337d90ef815a5acb3bf5e81ea77c8d46e85451ecb39a7418 WatchSource:0}: Error finding container aafccbecfe41446a337d90ef815a5acb3bf5e81ea77c8d46e85451ecb39a7418: Status 404 returned error can't find the container with id aafccbecfe41446a337d90ef815a5acb3bf5e81ea77c8d46e85451ecb39a7418 Jul 21 20:25:42.625593 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.625558 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rqtsp"] Jul 21 20:25:42.630489 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.630463 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rqtsp" Jul 21 20:25:42.639191 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.639162 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rbd47\"" Jul 21 20:25:42.648242 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.648221 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rqtsp"] Jul 21 20:25:42.697708 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.697604 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xlw\" (UniqueName: \"kubernetes.io/projected/81b9e0bf-9bee-4e89-bb00-5134a110609c-kube-api-access-s2xlw\") pod \"authorino-58cd955675-rqtsp\" (UID: \"81b9e0bf-9bee-4e89-bb00-5134a110609c\") " pod="kuadrant-system/authorino-58cd955675-rqtsp" Jul 21 20:25:42.798415 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.798378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xlw\" (UniqueName: \"kubernetes.io/projected/81b9e0bf-9bee-4e89-bb00-5134a110609c-kube-api-access-s2xlw\") pod \"authorino-58cd955675-rqtsp\" (UID: \"81b9e0bf-9bee-4e89-bb00-5134a110609c\") " pod="kuadrant-system/authorino-58cd955675-rqtsp" Jul 21 20:25:42.800826 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.800800 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" event={"ID":"62c1acb0-bd2e-49f5-866a-29fbb63ff00c","Type":"ContainerStarted","Data":"aafccbecfe41446a337d90ef815a5acb3bf5e81ea77c8d46e85451ecb39a7418"} Jul 21 20:25:42.810010 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.809986 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xlw\" (UniqueName: \"kubernetes.io/projected/81b9e0bf-9bee-4e89-bb00-5134a110609c-kube-api-access-s2xlw\") pod \"authorino-58cd955675-rqtsp\" (UID: \"81b9e0bf-9bee-4e89-bb00-5134a110609c\") " pod="kuadrant-system/authorino-58cd955675-rqtsp" Jul 21 20:25:42.939594 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:42.939560 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rqtsp" Jul 21 20:25:43.133825 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:43.133755 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rqtsp"] Jul 21 20:25:43.138460 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:25:43.138409 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b9e0bf_9bee_4e89_bb00_5134a110609c.slice/crio-4d232b1c1bebb6bbb3be2204a56afed2bc7152467b29525c2dd5a32229e649aa WatchSource:0}: Error finding container 4d232b1c1bebb6bbb3be2204a56afed2bc7152467b29525c2dd5a32229e649aa: Status 404 returned error can't find the container with id 4d232b1c1bebb6bbb3be2204a56afed2bc7152467b29525c2dd5a32229e649aa Jul 21 20:25:43.809912 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:43.809877 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rqtsp" event={"ID":"81b9e0bf-9bee-4e89-bb00-5134a110609c","Type":"ContainerStarted","Data":"4d232b1c1bebb6bbb3be2204a56afed2bc7152467b29525c2dd5a32229e649aa"} Jul 21 20:25:47.481926 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:47.481886 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rqtsp"] Jul 21 20:25:47.836548 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:47.836449 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" event={"ID":"62c1acb0-bd2e-49f5-866a-29fbb63ff00c","Type":"ContainerStarted","Data":"eb5100f6e57ae5b26d04946281ca723c5ffabe3c9a315cc5d23f878f9f0fc81d"} Jul 21 20:25:47.836722 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:47.836563 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:25:47.838162 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:47.838136 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rqtsp" event={"ID":"81b9e0bf-9bee-4e89-bb00-5134a110609c","Type":"ContainerStarted","Data":"e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321"} Jul 21 20:25:47.859233 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:47.859183 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" podStartSLOduration=1.600382145 podStartE2EDuration="5.859165264s" totalImagesPullingTime="4.258783119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:25:47.85623636 +0000 UTC m=+702.536362799" watchObservedRunningTime="2026-07-21 20:25:47.859165264 +0000 UTC m=+702.539291704" Jul 21 20:25:47.876510 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:47.876448 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rqtsp" podStartSLOduration=2.101306799 podStartE2EDuration="5.87643464s" totalImagesPullingTime="3.775127841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:25:47.875572471 +0000 UTC m=+702.555698907" watchObservedRunningTime="2026-07-21 20:25:47.87643464 +0000 UTC m=+702.556561076" Jul 21 20:25:48.843664 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:48.843627 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rqtsp" podUID="81b9e0bf-9bee-4e89-bb00-5134a110609c" containerName="authorino" containerID="cri-o://e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321" gracePeriod=30 Jul 21 20:25:49.099112 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.099043 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rqtsp" Jul 21 20:25:49.166109 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.166074 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2xlw\" (UniqueName: \"kubernetes.io/projected/81b9e0bf-9bee-4e89-bb00-5134a110609c-kube-api-access-s2xlw\") pod \"81b9e0bf-9bee-4e89-bb00-5134a110609c\" (UID: \"81b9e0bf-9bee-4e89-bb00-5134a110609c\") " Jul 21 20:25:49.168145 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.168117 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b9e0bf-9bee-4e89-bb00-5134a110609c-kube-api-access-s2xlw" (OuterVolumeSpecName: "kube-api-access-s2xlw") pod "81b9e0bf-9bee-4e89-bb00-5134a110609c" (UID: "81b9e0bf-9bee-4e89-bb00-5134a110609c"). InnerVolumeSpecName "kube-api-access-s2xlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:25:49.266703 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.266666 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xlw\" (UniqueName: \"kubernetes.io/projected/81b9e0bf-9bee-4e89-bb00-5134a110609c-kube-api-access-s2xlw\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:25:49.849572 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.849538 2567 generic.go:356] "Generic (PLEG): container finished" podID="81b9e0bf-9bee-4e89-bb00-5134a110609c" containerID="e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321" exitCode=0 Jul 21 20:25:49.850012 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.849585 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rqtsp" Jul 21 20:25:49.850012 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.849623 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rqtsp" event={"ID":"81b9e0bf-9bee-4e89-bb00-5134a110609c","Type":"ContainerDied","Data":"e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321"} Jul 21 20:25:49.850012 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.849663 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rqtsp" event={"ID":"81b9e0bf-9bee-4e89-bb00-5134a110609c","Type":"ContainerDied","Data":"4d232b1c1bebb6bbb3be2204a56afed2bc7152467b29525c2dd5a32229e649aa"} Jul 21 20:25:49.850012 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.849682 2567 scope.go:117] "RemoveContainer" containerID="e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321" Jul 21 20:25:49.858221 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.858208 2567 scope.go:117] "RemoveContainer" containerID="e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321" Jul 21 20:25:49.858489 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:25:49.858471 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321\": container with ID starting with e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321 not found: ID does not exist" containerID="e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321" Jul 21 20:25:49.858565 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.858502 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321"} err="failed to get container status \"e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321\": rpc error: code = NotFound desc = could not find container \"e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321\": container with ID starting with e51733d98642282e8a40c73ba07702b0c13dbc062d3a5792b8c91e0342356321 not found: ID does not exist" Jul 21 20:25:49.870641 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.870618 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rqtsp"] Jul 21 20:25:49.874191 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:49.874169 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rqtsp"] Jul 21 20:25:51.804731 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:51.804695 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b9e0bf-9bee-4e89-bb00-5134a110609c" path="/var/lib/kubelet/pods/81b9e0bf-9bee-4e89-bb00-5134a110609c/volumes" Jul 21 20:25:58.844771 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:25:58.844693 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-bh898" Jul 21 20:26:16.576658 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.576621 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2ljqw"] Jul 21 20:26:16.577138 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.577027 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b9e0bf-9bee-4e89-bb00-5134a110609c" containerName="authorino" Jul 21 20:26:16.577138 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.577039 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b9e0bf-9bee-4e89-bb00-5134a110609c" containerName="authorino" Jul 21 20:26:16.577138 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.577093 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b9e0bf-9bee-4e89-bb00-5134a110609c" containerName="authorino" Jul 21 20:26:16.581368 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.581350 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" Jul 21 20:26:16.589228 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.589211 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rbd47\"" Jul 21 20:26:16.593159 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.593134 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2ljqw"] Jul 21 20:26:16.606794 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.606771 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4v8q\" (UniqueName: \"kubernetes.io/projected/2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba-kube-api-access-b4v8q\") pod \"authorino-5d4b67c446-2ljqw\" (UID: \"2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba\") " pod="kuadrant-system/authorino-5d4b67c446-2ljqw" Jul 21 20:26:16.707439 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.707401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4v8q\" (UniqueName: \"kubernetes.io/projected/2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba-kube-api-access-b4v8q\") pod \"authorino-5d4b67c446-2ljqw\" (UID: \"2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba\") " pod="kuadrant-system/authorino-5d4b67c446-2ljqw" Jul 21 20:26:16.716932 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.716898 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4v8q\" (UniqueName: \"kubernetes.io/projected/2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba-kube-api-access-b4v8q\") pod \"authorino-5d4b67c446-2ljqw\" (UID: \"2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba\") " pod="kuadrant-system/authorino-5d4b67c446-2ljqw" Jul 21 20:26:16.826282 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.826252 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2ljqw"] Jul 21 20:26:16.826523 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.826511 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" Jul 21 20:26:16.866685 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.866651 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d6c6ccc54-l6gzh"] Jul 21 20:26:16.870255 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.870229 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" Jul 21 20:26:16.886957 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.886913 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d6c6ccc54-l6gzh"] Jul 21 20:26:16.909708 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.909678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfddk\" (UniqueName: \"kubernetes.io/projected/32ab45d5-858a-41ac-91ac-088036309c3c-kube-api-access-bfddk\") pod \"authorino-6d6c6ccc54-l6gzh\" (UID: \"32ab45d5-858a-41ac-91ac-088036309c3c\") " pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" Jul 21 20:26:16.972268 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.972239 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2ljqw"] Jul 21 20:26:16.974654 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:26:16.974621 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c255fc9_ffaf_4b2d_a260_7c5f4ac4a0ba.slice/crio-cc87bf832b6d633e44262a96bd818357a0c0a0921dbdb88883a8e4ba1ac02b6c WatchSource:0}: Error finding container cc87bf832b6d633e44262a96bd818357a0c0a0921dbdb88883a8e4ba1ac02b6c: Status 404 returned error can't find the container with id cc87bf832b6d633e44262a96bd818357a0c0a0921dbdb88883a8e4ba1ac02b6c Jul 21 20:26:16.997250 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:16.997219 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" event={"ID":"2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba","Type":"ContainerStarted","Data":"cc87bf832b6d633e44262a96bd818357a0c0a0921dbdb88883a8e4ba1ac02b6c"} Jul 21 20:26:17.010655 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:17.010627 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfddk\" (UniqueName: \"kubernetes.io/projected/32ab45d5-858a-41ac-91ac-088036309c3c-kube-api-access-bfddk\") pod \"authorino-6d6c6ccc54-l6gzh\" (UID: \"32ab45d5-858a-41ac-91ac-088036309c3c\") " pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" Jul 21 20:26:17.023436 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:17.023406 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfddk\" (UniqueName: \"kubernetes.io/projected/32ab45d5-858a-41ac-91ac-088036309c3c-kube-api-access-bfddk\") pod \"authorino-6d6c6ccc54-l6gzh\" (UID: \"32ab45d5-858a-41ac-91ac-088036309c3c\") " pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" Jul 21 20:26:17.145701 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:17.145660 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d6c6ccc54-l6gzh"] Jul 21 20:26:17.146011 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:17.145994 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" Jul 21 20:26:17.292755 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:17.292731 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d6c6ccc54-l6gzh"] Jul 21 20:26:17.294671 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:26:17.294645 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32ab45d5_858a_41ac_91ac_088036309c3c.slice/crio-24aaaf978ac59104e851a8c0ce47efb059155763c484f5c23d8fee4575a85bbf WatchSource:0}: Error finding container 24aaaf978ac59104e851a8c0ce47efb059155763c484f5c23d8fee4575a85bbf: Status 404 returned error can't find the container with id 24aaaf978ac59104e851a8c0ce47efb059155763c484f5c23d8fee4575a85bbf Jul 21 20:26:18.003723 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.003636 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" event={"ID":"32ab45d5-858a-41ac-91ac-088036309c3c","Type":"ContainerStarted","Data":"c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14"} Jul 21 20:26:18.003723 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.003680 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" podUID="32ab45d5-858a-41ac-91ac-088036309c3c" containerName="authorino" containerID="cri-o://c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14" gracePeriod=30 Jul 21 20:26:18.004232 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.003688 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" event={"ID":"32ab45d5-858a-41ac-91ac-088036309c3c","Type":"ContainerStarted","Data":"24aaaf978ac59104e851a8c0ce47efb059155763c484f5c23d8fee4575a85bbf"} Jul 21 20:26:18.005316 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.005293 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" event={"ID":"2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba","Type":"ContainerStarted","Data":"40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4"} Jul 21 20:26:18.005420 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.005376 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" podUID="2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba" containerName="authorino" containerID="cri-o://40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4" gracePeriod=30 Jul 21 20:26:18.047203 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.047162 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" podStartSLOduration=1.6434456769999999 podStartE2EDuration="2.047147561s" totalImagesPullingTime="403.701884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:26:18.046024829 +0000 UTC m=+732.726151290" watchObservedRunningTime="2026-07-21 20:26:18.047147561 +0000 UTC m=+732.727273997" Jul 21 20:26:18.078826 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.078776 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" podStartSLOduration=1.6719208509999999 podStartE2EDuration="2.078757897s" totalImagesPullingTime="406.837046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:26:18.074832314 +0000 UTC m=+732.754958759" watchObservedRunningTime="2026-07-21 20:26:18.078757897 +0000 UTC m=+732.758884333" Jul 21 20:26:18.276482 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.276459 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" Jul 21 20:26:18.279098 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.279081 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" Jul 21 20:26:18.322876 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.322845 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfddk\" (UniqueName: \"kubernetes.io/projected/32ab45d5-858a-41ac-91ac-088036309c3c-kube-api-access-bfddk\") pod \"32ab45d5-858a-41ac-91ac-088036309c3c\" (UID: \"32ab45d5-858a-41ac-91ac-088036309c3c\") " Jul 21 20:26:18.323088 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.322992 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4v8q\" (UniqueName: \"kubernetes.io/projected/2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba-kube-api-access-b4v8q\") pod \"2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba\" (UID: \"2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba\") " Jul 21 20:26:18.325343 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.325312 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ab45d5-858a-41ac-91ac-088036309c3c-kube-api-access-bfddk" (OuterVolumeSpecName: "kube-api-access-bfddk") pod "32ab45d5-858a-41ac-91ac-088036309c3c" (UID: "32ab45d5-858a-41ac-91ac-088036309c3c"). InnerVolumeSpecName "kube-api-access-bfddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:26:18.325480 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.325461 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba-kube-api-access-b4v8q" (OuterVolumeSpecName: "kube-api-access-b4v8q") pod "2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba" (UID: "2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba"). InnerVolumeSpecName "kube-api-access-b4v8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 20:26:18.423874 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.423836 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4v8q\" (UniqueName: \"kubernetes.io/projected/2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba-kube-api-access-b4v8q\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:26:18.423874 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:18.423867 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfddk\" (UniqueName: \"kubernetes.io/projected/32ab45d5-858a-41ac-91ac-088036309c3c-kube-api-access-bfddk\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 20:26:19.011871 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.011792 2567 generic.go:356] "Generic (PLEG): container finished" podID="32ab45d5-858a-41ac-91ac-088036309c3c" containerID="c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14" exitCode=0 Jul 21 20:26:19.011871 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.011848 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" Jul 21 20:26:19.012399 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.011884 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" event={"ID":"32ab45d5-858a-41ac-91ac-088036309c3c","Type":"ContainerDied","Data":"c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14"} Jul 21 20:26:19.012399 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.011933 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6c6ccc54-l6gzh" event={"ID":"32ab45d5-858a-41ac-91ac-088036309c3c","Type":"ContainerDied","Data":"24aaaf978ac59104e851a8c0ce47efb059155763c484f5c23d8fee4575a85bbf"} Jul 21 20:26:19.012399 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.011971 2567 scope.go:117] "RemoveContainer" containerID="c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14" Jul 21 20:26:19.013572 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.013551 2567 generic.go:356] "Generic (PLEG): container finished" podID="2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba" containerID="40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4" exitCode=0 Jul 21 20:26:19.013658 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.013588 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" event={"ID":"2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba","Type":"ContainerDied","Data":"40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4"} Jul 21 20:26:19.013658 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.013605 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" Jul 21 20:26:19.013658 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.013617 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2ljqw" event={"ID":"2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba","Type":"ContainerDied","Data":"cc87bf832b6d633e44262a96bd818357a0c0a0921dbdb88883a8e4ba1ac02b6c"} Jul 21 20:26:19.023017 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.022998 2567 scope.go:117] "RemoveContainer" containerID="c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14" Jul 21 20:26:19.023343 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:26:19.023324 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14\": container with ID starting with c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14 not found: ID does not exist" containerID="c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14" Jul 21 20:26:19.023428 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.023354 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14"} err="failed to get container status \"c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14\": rpc error: code = NotFound desc = could not find container \"c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14\": container with ID starting with c7a704e7f7e709cc336b80aca0261d09cee6e7d648ee0f4360775ae94e3f2f14 not found: ID does not exist" Jul 21 20:26:19.023428 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.023379 2567 scope.go:117] "RemoveContainer" containerID="40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4" Jul 21 20:26:19.031678 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.031657 2567 scope.go:117] "RemoveContainer" containerID="40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4" Jul 21 20:26:19.031960 ip-10-0-141-132 kubenswrapper[2567]: E0721 20:26:19.031929 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4\": container with ID starting with 40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4 not found: ID does not exist" containerID="40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4" Jul 21 20:26:19.032012 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.031971 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4"} err="failed to get container status \"40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4\": rpc error: code = NotFound desc = could not find container \"40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4\": container with ID starting with 40fe3af8336a77ea37b1e1c5881f7af3d9435f7734acd41ea8936c4d7c6c79b4 not found: ID does not exist" Jul 21 20:26:19.054864 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.054840 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d6c6ccc54-l6gzh"] Jul 21 20:26:19.062922 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.062900 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d6c6ccc54-l6gzh"] Jul 21 20:26:19.110239 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.110205 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2ljqw"] Jul 21 20:26:19.113533 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.113508 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2ljqw"] Jul 21 20:26:19.806023 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.805991 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba" path="/var/lib/kubelet/pods/2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba/volumes" Jul 21 20:26:19.806409 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:19.806389 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ab45d5-858a-41ac-91ac-088036309c3c" path="/var/lib/kubelet/pods/32ab45d5-858a-41ac-91ac-088036309c3c/volumes" Jul 21 20:26:58.645704 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:26:58.645658 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:27:12.903287 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:12.903252 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:27:23.109002 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.108926 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf"] Jul 21 20:27:23.109395 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.109330 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba" containerName="authorino" Jul 21 20:27:23.109395 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.109342 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba" containerName="authorino" Jul 21 20:27:23.109395 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.109354 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ab45d5-858a-41ac-91ac-088036309c3c" containerName="authorino" Jul 21 20:27:23.109395 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.109360 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ab45d5-858a-41ac-91ac-088036309c3c" containerName="authorino" Jul 21 20:27:23.109589 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.109417 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c255fc9-ffaf-4b2d-a260-7c5f4ac4a0ba" containerName="authorino" Jul 21 20:27:23.109589 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.109427 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ab45d5-858a-41ac-91ac-088036309c3c" containerName="authorino" Jul 21 20:27:23.113839 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.113821 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.116628 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.116606 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 20:27:23.116910 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.116893 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sn7s4\"" Jul 21 20:27:23.117346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.117325 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 20:27:23.121851 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.121833 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 20:27:23.126810 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.126781 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf"] Jul 21 20:27:23.194480 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.194453 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.194641 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.194487 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6v55\" (UniqueName: \"kubernetes.io/projected/5c628870-82ea-42e7-b807-a28c5c1ff7e1-kube-api-access-d6v55\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.194641 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.194527 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c628870-82ea-42e7-b807-a28c5c1ff7e1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.194641 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.194587 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.194641 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.194622 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.194812 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.194652 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.295749 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.295681 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c628870-82ea-42e7-b807-a28c5c1ff7e1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.295749 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.295727 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.296055 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.295874 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.296055 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.295937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.296178 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.296063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.296178 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.296094 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v55\" (UniqueName: \"kubernetes.io/projected/5c628870-82ea-42e7-b807-a28c5c1ff7e1-kube-api-access-d6v55\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.296284 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.296263 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.296331 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.296303 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.296431 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.296411 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.298072 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.298050 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c628870-82ea-42e7-b807-a28c5c1ff7e1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.298314 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.298298 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c628870-82ea-42e7-b807-a28c5c1ff7e1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.308744 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.308723 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v55\" (UniqueName: \"kubernetes.io/projected/5c628870-82ea-42e7-b807-a28c5c1ff7e1-kube-api-access-d6v55\") pod \"e2e-distinct-simulated-kserve-59975d69f9-cqltf\" (UID: \"5c628870-82ea-42e7-b807-a28c5c1ff7e1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.424407 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.424374 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:23.564284 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:23.564249 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf"] Jul 21 20:27:23.567388 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:27:23.567363 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c628870_82ea_42e7_b807_a28c5c1ff7e1.slice/crio-b20804312bf64790c46b4ab563eaa1309ed4a741870b1f73e8ad5b921e82d8be WatchSource:0}: Error finding container b20804312bf64790c46b4ab563eaa1309ed4a741870b1f73e8ad5b921e82d8be: Status 404 returned error can't find the container with id b20804312bf64790c46b4ab563eaa1309ed4a741870b1f73e8ad5b921e82d8be Jul 21 20:27:24.000726 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:24.000058 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:27:24.363007 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:24.362905 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" event={"ID":"5c628870-82ea-42e7-b807-a28c5c1ff7e1","Type":"ContainerStarted","Data":"b20804312bf64790c46b4ab563eaa1309ed4a741870b1f73e8ad5b921e82d8be"} Jul 21 20:27:25.369980 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:25.369923 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" event={"ID":"5c628870-82ea-42e7-b807-a28c5c1ff7e1","Type":"ContainerStarted","Data":"f24c44925e912fca194cb66952e03d8dcf00aa32f375ecc3d9000bb903c24e4a"} Jul 21 20:27:25.389243 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:25.389193 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" podStartSLOduration=1.32820427 podStartE2EDuration="2.389177323s" totalImagesPullingTime="1.060973053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:27:25.38753278 +0000 UTC m=+800.067659214" watchObservedRunningTime="2026-07-21 20:27:25.389177323 +0000 UTC m=+800.069303812" Jul 21 20:27:33.424676 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:33.424577 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:33.424676 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:33.424659 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:33.437318 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:33.437290 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:34.430421 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:34.430390 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-cqltf" Jul 21 20:27:35.607815 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:35.607779 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:27:39.181611 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.181572 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2"] Jul 21 20:27:39.185246 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.185225 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.188275 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.188259 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 21 20:27:39.197452 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.197421 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2"] Jul 21 20:27:39.343324 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.343271 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2mkc\" (UniqueName: \"kubernetes.io/projected/97d3a1e6-773c-4d14-819e-f27235042c2f-kube-api-access-t2mkc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.343324 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.343324 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.343557 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.343384 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d3a1e6-773c-4d14-819e-f27235042c2f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.343557 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.343409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.343557 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.343491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.343557 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.343522 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.444108 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.444015 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d3a1e6-773c-4d14-819e-f27235042c2f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.444108 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.444057 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.444346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.444168 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.444346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.444188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.444346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.444242 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2mkc\" (UniqueName: \"kubernetes.io/projected/97d3a1e6-773c-4d14-819e-f27235042c2f-kube-api-access-t2mkc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.444346 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.444260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.444634 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.444609 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.444762 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.444629 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.444762 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.444680 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.446643 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.446621 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97d3a1e6-773c-4d14-819e-f27235042c2f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.446790 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.446774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97d3a1e6-773c-4d14-819e-f27235042c2f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.453135 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.453113 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2mkc\" (UniqueName: \"kubernetes.io/projected/97d3a1e6-773c-4d14-819e-f27235042c2f-kube-api-access-t2mkc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2\" (UID: \"97d3a1e6-773c-4d14-819e-f27235042c2f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.496183 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.496149 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:39.632000 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:39.631970 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2"] Jul 21 20:27:39.633311 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:27:39.633286 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d3a1e6_773c_4d14_819e_f27235042c2f.slice/crio-4035305bd8d0e5362666b68886b7e8987bfbe5ca2f16b614eb56e40c845826f8 WatchSource:0}: Error finding container 4035305bd8d0e5362666b68886b7e8987bfbe5ca2f16b614eb56e40c845826f8: Status 404 returned error can't find the container with id 4035305bd8d0e5362666b68886b7e8987bfbe5ca2f16b614eb56e40c845826f8 Jul 21 20:27:40.412202 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:40.412166 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:27:40.457995 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:40.457962 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" event={"ID":"97d3a1e6-773c-4d14-819e-f27235042c2f","Type":"ContainerStarted","Data":"338ff89e66aa23f1e9147d72b460d3376be94cd3e641b22678f9d17884b6f176"} Jul 21 20:27:40.458158 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:40.458023 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" event={"ID":"97d3a1e6-773c-4d14-819e-f27235042c2f","Type":"ContainerStarted","Data":"4035305bd8d0e5362666b68886b7e8987bfbe5ca2f16b614eb56e40c845826f8"} Jul 21 20:27:40.484050 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:40.483999 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" podStartSLOduration=1.274940602 podStartE2EDuration="1.483984273s" totalImagesPullingTime="209.043671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:27:40.482224101 +0000 UTC m=+815.162350552" watchObservedRunningTime="2026-07-21 20:27:40.483984273 +0000 UTC m=+815.164110711" Jul 21 20:27:49.496964 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:49.496902 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:49.497417 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:49.497009 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:49.509911 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:49.509885 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:27:49.521067 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:27:49.521045 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2" Jul 21 20:28:01.000352 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:28:01.000315 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:28:28.295867 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:28:28.295830 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:29:05.750570 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:05.750544 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:29:05.753241 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:05.753221 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:29:29.518838 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.518807 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79979d5c99-hzkmt"] Jul 21 20:29:29.522168 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.522146 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79979d5c99-hzkmt" Jul 21 20:29:29.527276 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.527259 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 20:29:29.527276 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.527269 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rbd47\"" Jul 21 20:29:29.540655 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.540634 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79979d5c99-hzkmt"] Jul 21 20:29:29.652496 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.652462 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdf0d01d-f88a-40d3-bfa2-3f862001eb64-tls-cert\") pod \"authorino-79979d5c99-hzkmt\" (UID: \"cdf0d01d-f88a-40d3-bfa2-3f862001eb64\") " pod="kuadrant-system/authorino-79979d5c99-hzkmt" Jul 21 20:29:29.652664 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.652531 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrxc2\" (UniqueName: \"kubernetes.io/projected/cdf0d01d-f88a-40d3-bfa2-3f862001eb64-kube-api-access-nrxc2\") pod \"authorino-79979d5c99-hzkmt\" (UID: \"cdf0d01d-f88a-40d3-bfa2-3f862001eb64\") " pod="kuadrant-system/authorino-79979d5c99-hzkmt" Jul 21 20:29:29.753265 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.753232 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdf0d01d-f88a-40d3-bfa2-3f862001eb64-tls-cert\") pod \"authorino-79979d5c99-hzkmt\" (UID: \"cdf0d01d-f88a-40d3-bfa2-3f862001eb64\") " pod="kuadrant-system/authorino-79979d5c99-hzkmt" Jul 21 20:29:29.753422 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.753301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrxc2\" (UniqueName: \"kubernetes.io/projected/cdf0d01d-f88a-40d3-bfa2-3f862001eb64-kube-api-access-nrxc2\") pod \"authorino-79979d5c99-hzkmt\" (UID: \"cdf0d01d-f88a-40d3-bfa2-3f862001eb64\") " pod="kuadrant-system/authorino-79979d5c99-hzkmt" Jul 21 20:29:29.755692 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.755657 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdf0d01d-f88a-40d3-bfa2-3f862001eb64-tls-cert\") pod \"authorino-79979d5c99-hzkmt\" (UID: \"cdf0d01d-f88a-40d3-bfa2-3f862001eb64\") " pod="kuadrant-system/authorino-79979d5c99-hzkmt" Jul 21 20:29:29.767371 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.767351 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrxc2\" (UniqueName: \"kubernetes.io/projected/cdf0d01d-f88a-40d3-bfa2-3f862001eb64-kube-api-access-nrxc2\") pod \"authorino-79979d5c99-hzkmt\" (UID: \"cdf0d01d-f88a-40d3-bfa2-3f862001eb64\") " pod="kuadrant-system/authorino-79979d5c99-hzkmt" Jul 21 20:29:29.831475 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.831402 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79979d5c99-hzkmt" Jul 21 20:29:29.986972 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.986911 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79979d5c99-hzkmt"] Jul 21 20:29:29.988433 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:29:29.988405 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdf0d01d_f88a_40d3_bfa2_3f862001eb64.slice/crio-904e6e253eeb66cfe3f07e41a834f7a8029db4b6062328e07fe80c3ae35b4a21 WatchSource:0}: Error finding container 904e6e253eeb66cfe3f07e41a834f7a8029db4b6062328e07fe80c3ae35b4a21: Status 404 returned error can't find the container with id 904e6e253eeb66cfe3f07e41a834f7a8029db4b6062328e07fe80c3ae35b4a21 Jul 21 20:29:29.989678 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:29.989662 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 20:29:30.056784 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:30.056748 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79979d5c99-hzkmt" event={"ID":"cdf0d01d-f88a-40d3-bfa2-3f862001eb64","Type":"ContainerStarted","Data":"904e6e253eeb66cfe3f07e41a834f7a8029db4b6062328e07fe80c3ae35b4a21"} Jul 21 20:29:31.064166 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:31.064079 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79979d5c99-hzkmt" event={"ID":"cdf0d01d-f88a-40d3-bfa2-3f862001eb64","Type":"ContainerStarted","Data":"94a19d736335b23ea67ebbafaae6c96b0b1ac34e1e05ab67ac49d21050968044"} Jul 21 20:29:31.118082 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:29:31.118028 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79979d5c99-hzkmt" podStartSLOduration=1.384260727 podStartE2EDuration="2.118007077s" totalImagesPullingTime="733.74635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:29:31.113544205 +0000 UTC m=+925.793670663" watchObservedRunningTime="2026-07-21 20:29:31.118007077 +0000 UTC m=+925.798133518" Jul 21 20:30:00.588880 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:30:00.588842 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:30:10.296919 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:30:10.296882 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:30:21.100803 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:30:21.100768 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:32:01.594063 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:32:01.593974 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:32:13.087075 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:32:13.087038 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:32:48.787771 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:32:48.787733 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:32:58.289126 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:32:58.289091 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:33:08.695067 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:33:08.695031 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:33:16.288869 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:33:16.288830 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:33:27.720584 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:33:27.720503 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:33:37.197693 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:33:37.197654 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:33:48.903213 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:33:48.903180 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:33:59.333628 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:33:59.333590 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:34:05.787239 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:34:05.787212 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:34:05.790514 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:34:05.790493 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:34:21.095828 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:34:21.095788 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:34:33.270006 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:34:33.269964 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:35:01.230068 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:35:01.229992 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:35:12.606720 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:35:12.606683 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:35:28.593169 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:35:28.593135 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:35:40.505008 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:35:40.504965 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:35:55.491868 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:35:55.491837 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:37:57.488629 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:37:57.488544 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:38:09.784555 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:38:09.784515 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:38:19.152575 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:38:19.152529 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:38:33.134960 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:38:33.134898 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:38:42.783881 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:38:42.783835 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:38:52.091640 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:38:52.091602 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:39:01.500762 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:39:01.500719 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:39:05.824484 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:39:05.824455 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:39:05.829148 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:39:05.829126 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:39:12.785542 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:39:12.785508 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:39:22.390803 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:39:22.390765 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:39:32.293248 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:39:32.293163 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:39:44.383611 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:39:44.383577 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:40:09.195230 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:40:09.195179 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:40:17.988168 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:40:17.988130 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:40:30.020270 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:40:30.020231 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:40:53.793014 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:40:53.792974 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:41:14.599121 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:41:14.599023 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:41:30.521017 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:41:30.520981 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:41:41.104132 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:41:41.104097 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:41:53.399188 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:41:53.399157 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:42:09.095211 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:42:09.095179 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:42:18.284231 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:42:18.284189 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:42:31.888463 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:42:31.888369 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:42:41.185444 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:42:41.185407 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:42:51.712180 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:42:51.712136 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:43:04.094156 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:43:04.094117 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:43:18.904808 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:43:18.904769 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:43:29.386433 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:43:29.386400 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:43:42.284542 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:43:42.284500 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:43:50.285485 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:43:50.285446 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:44:02.435380 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:44:02.435299 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:44:05.859888 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:44:05.859863 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:44:05.866309 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:44:05.866289 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:44:13.010191 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:44:13.010156 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:44:26.111671 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:44:26.111634 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:44:36.137780 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:44:36.137740 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:44:59.112852 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:44:59.112811 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:45:10.127716 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:45:10.127680 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:45:36.810321 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:45:36.810224 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:45:49.100590 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:45:49.100551 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:46:05.485951 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:46:05.485911 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:46:18.083927 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:46:18.083887 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:48:37.391763 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:48:37.391680 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:48:50.004756 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:48:50.004718 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:49:05.896630 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:49:05.896603 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:49:05.904027 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:49:05.904006 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:49:06.334250 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:49:06.334170 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:49:18.799934 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:49:18.799897 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:50:26.016321 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:50:26.016280 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:50:35.094026 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:50:35.093992 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:50:46.184712 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:50:46.184673 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:51:05.387051 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:51:05.387020 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:51:28.287925 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:51:28.287839 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:51:57.401923 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:51:57.401883 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:52:23.092103 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:52:23.092069 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:54:05.933180 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:54:05.933152 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:54:05.941524 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:54:05.941500 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:55:36.188518 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:55:36.188483 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:56:19.884988 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:56:19.884956 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:56:42.217126 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:56:42.217091 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:56:58.290079 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:56:58.290038 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:57:37.470865 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.470783 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx"] Jul 21 20:57:37.474457 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.474424 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.477078 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.477054 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 20:57:37.477192 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.477075 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zjkdf\"" Jul 21 20:57:37.478045 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.478029 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2ba36e-serving-cert\"" Jul 21 20:57:37.480921 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.480903 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 21 20:57:37.495456 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.495431 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx"] Jul 21 20:57:37.653450 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.653416 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6fb955-805e-40c1-ac1b-0d996d196476-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx\" (UID: \"cb6fb955-805e-40c1-ac1b-0d996d196476\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.653450 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.653456 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zj9\" (UniqueName: \"kubernetes.io/projected/cb6fb955-805e-40c1-ac1b-0d996d196476-kube-api-access-k4zj9\") pod \"maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx\" (UID: \"cb6fb955-805e-40c1-ac1b-0d996d196476\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.653714 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.653564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6fb955-805e-40c1-ac1b-0d996d196476-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx\" (UID: \"cb6fb955-805e-40c1-ac1b-0d996d196476\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.755195 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.755099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6fb955-805e-40c1-ac1b-0d996d196476-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx\" (UID: \"cb6fb955-805e-40c1-ac1b-0d996d196476\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.755195 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.755189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6fb955-805e-40c1-ac1b-0d996d196476-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx\" (UID: \"cb6fb955-805e-40c1-ac1b-0d996d196476\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.755452 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.755218 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zj9\" (UniqueName: \"kubernetes.io/projected/cb6fb955-805e-40c1-ac1b-0d996d196476-kube-api-access-k4zj9\") pod \"maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx\" (UID: \"cb6fb955-805e-40c1-ac1b-0d996d196476\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.755914 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.755890 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6fb955-805e-40c1-ac1b-0d996d196476-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx\" (UID: \"cb6fb955-805e-40c1-ac1b-0d996d196476\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.757777 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.757750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb6fb955-805e-40c1-ac1b-0d996d196476-maas-api-tls\") pod \"maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx\" (UID: \"cb6fb955-805e-40c1-ac1b-0d996d196476\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.770210 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.770188 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zj9\" (UniqueName: \"kubernetes.io/projected/cb6fb955-805e-40c1-ac1b-0d996d196476-kube-api-access-k4zj9\") pod \"maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx\" (UID: \"cb6fb955-805e-40c1-ac1b-0d996d196476\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.786853 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.786834 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:37.935622 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.935592 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx"] Jul 21 20:57:37.936536 ip-10-0-141-132 kubenswrapper[2567]: W0721 20:57:37.936505 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6fb955_805e_40c1_ac1b_0d996d196476.slice/crio-6c83e01fe86329cca1e2b7269f2c002e796b1b435ee7c5f312ef30c75c03fc0a WatchSource:0}: Error finding container 6c83e01fe86329cca1e2b7269f2c002e796b1b435ee7c5f312ef30c75c03fc0a: Status 404 returned error can't find the container with id 6c83e01fe86329cca1e2b7269f2c002e796b1b435ee7c5f312ef30c75c03fc0a Jul 21 20:57:37.938035 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:37.938018 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 20:57:38.309417 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:38.309384 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" event={"ID":"cb6fb955-805e-40c1-ac1b-0d996d196476","Type":"ContainerStarted","Data":"6c83e01fe86329cca1e2b7269f2c002e796b1b435ee7c5f312ef30c75c03fc0a"} Jul 21 20:57:40.321907 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:40.321869 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" event={"ID":"cb6fb955-805e-40c1-ac1b-0d996d196476","Type":"ContainerStarted","Data":"adb42cf60996ed64c52cd1e2dd7749114d97dcc9b6d24f31d4437963f46630cb"} Jul 21 20:57:40.322333 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:40.322076 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:40.341255 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:40.341178 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" podStartSLOduration=1.139563836 podStartE2EDuration="3.341163194s" totalImagesPullingTime="2.201599358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 20:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 20:57:40.338831225 +0000 UTC m=+2615.018957662" watchObservedRunningTime="2026-07-21 20:57:40.341163194 +0000 UTC m=+2615.021289630" Jul 21 20:57:46.333915 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:46.333886 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx" Jul 21 20:57:48.018735 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:57:48.018700 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:58:06.751428 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:58:06.751389 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:58:33.322574 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:58:33.322533 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:58:49.297436 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:58:49.297399 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:59:05.969384 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:59:05.969354 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:59:05.979624 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:59:05.979597 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 20:59:16.288001 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:59:16.287964 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 20:59:59.106853 ip-10-0-141-132 kubenswrapper[2567]: I0721 20:59:59.106816 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:00:00.221532 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.221485 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2"] Jul 21 21:00:00.225229 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.225209 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj"] Jul 21 21:00:00.225382 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.225329 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2" Jul 21 21:00:00.228268 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.228241 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kvpgn\"" Jul 21 21:00:00.228459 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.228445 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj" Jul 21 21:00:00.239088 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.237008 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2" Jul 21 21:00:00.241695 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.241318 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj"] Jul 21 21:00:00.244934 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.244910 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2"] Jul 21 21:00:00.248456 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.248433 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj" Jul 21 21:00:00.603618 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.603592 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2"] Jul 21 21:00:00.606586 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:00:00.606557 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435b529f_0961_4e4f_92da_3db61135f8ab.slice/crio-32ff310ad3a628db59ad4aeabd96cdb3a0997bd4ede17c93fd765feffa255fb8 WatchSource:0}: Error finding container 32ff310ad3a628db59ad4aeabd96cdb3a0997bd4ede17c93fd765feffa255fb8: Status 404 returned error can't find the container with id 32ff310ad3a628db59ad4aeabd96cdb3a0997bd4ede17c93fd765feffa255fb8 Jul 21 21:00:00.632288 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:00.632267 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj"] Jul 21 21:00:00.634135 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:00:00.634112 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8862c886_7e33_4048_9061_6decc23738be.slice/crio-a88e07dbdd8bba4cf3b295a016b18683da73cde45fb29b63c5b9a1a836fb5a6b WatchSource:0}: Error finding container a88e07dbdd8bba4cf3b295a016b18683da73cde45fb29b63c5b9a1a836fb5a6b: Status 404 returned error can't find the container with id a88e07dbdd8bba4cf3b295a016b18683da73cde45fb29b63c5b9a1a836fb5a6b Jul 21 21:00:01.105545 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:01.105513 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj" event={"ID":"8862c886-7e33-4048-9061-6decc23738be","Type":"ContainerStarted","Data":"a88e07dbdd8bba4cf3b295a016b18683da73cde45fb29b63c5b9a1a836fb5a6b"} Jul 21 21:00:01.106660 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:01.106633 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2" event={"ID":"435b529f-0961-4e4f-92da-3db61135f8ab","Type":"ContainerStarted","Data":"32ff310ad3a628db59ad4aeabd96cdb3a0997bd4ede17c93fd765feffa255fb8"} Jul 21 21:00:06.137647 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:06.137613 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj" event={"ID":"8862c886-7e33-4048-9061-6decc23738be","Type":"ContainerStarted","Data":"95ba086f746325c3f2ecf10e8f591d4850bd5ddb5583f79ccdcf54aa1363ad29"} Jul 21 21:00:06.159836 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:06.159745 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj" podStartSLOduration=0.878638897 podStartE2EDuration="6.159729981s" totalImagesPullingTime="5.281091084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:06.158038522 +0000 UTC m=+2760.838164960" watchObservedRunningTime="2026-07-21 21:00:06.159729981 +0000 UTC m=+2760.839856415" Jul 21 21:00:07.145040 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:07.145005 2567 generic.go:356] "Generic (PLEG): container finished" podID="435b529f-0961-4e4f-92da-3db61135f8ab" containerID="acb4a66da6087d663bb347f15a916e5103026a02031df1d978f0e8d02560de7f" exitCode=0 Jul 21 21:00:07.145488 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:07.145103 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2" event={"ID":"435b529f-0961-4e4f-92da-3db61135f8ab","Type":"ContainerDied","Data":"acb4a66da6087d663bb347f15a916e5103026a02031df1d978f0e8d02560de7f"} Jul 21 21:00:07.146664 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:07.146642 2567 generic.go:356] "Generic (PLEG): container finished" podID="8862c886-7e33-4048-9061-6decc23738be" containerID="95ba086f746325c3f2ecf10e8f591d4850bd5ddb5583f79ccdcf54aa1363ad29" exitCode=0 Jul 21 21:00:07.146769 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:07.146724 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj" event={"ID":"8862c886-7e33-4048-9061-6decc23738be","Type":"ContainerDied","Data":"95ba086f746325c3f2ecf10e8f591d4850bd5ddb5583f79ccdcf54aa1363ad29"} Jul 21 21:00:08.285534 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:08.285505 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj" Jul 21 21:00:08.320358 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:08.320335 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2" Jul 21 21:00:09.159584 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:09.159556 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj" Jul 21 21:00:09.159797 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:09.159599 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj" event={"ID":"8862c886-7e33-4048-9061-6decc23738be","Type":"ContainerDied","Data":"a88e07dbdd8bba4cf3b295a016b18683da73cde45fb29b63c5b9a1a836fb5a6b"} Jul 21 21:00:09.159797 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:09.159650 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a88e07dbdd8bba4cf3b295a016b18683da73cde45fb29b63c5b9a1a836fb5a6b" Jul 21 21:00:09.161012 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:09.160983 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2" event={"ID":"435b529f-0961-4e4f-92da-3db61135f8ab","Type":"ContainerDied","Data":"32ff310ad3a628db59ad4aeabd96cdb3a0997bd4ede17c93fd765feffa255fb8"} Jul 21 21:00:09.161142 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:09.161026 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32ff310ad3a628db59ad4aeabd96cdb3a0997bd4ede17c93fd765feffa255fb8" Jul 21 21:00:09.161142 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:09.160992 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2" Jul 21 21:00:13.310903 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.310874 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d"] Jul 21 21:00:13.311301 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.311265 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8862c886-7e33-4048-9061-6decc23738be" containerName="cleanup" Jul 21 21:00:13.311301 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.311276 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8862c886-7e33-4048-9061-6decc23738be" containerName="cleanup" Jul 21 21:00:13.311301 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.311295 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435b529f-0961-4e4f-92da-3db61135f8ab" containerName="cleanup" Jul 21 21:00:13.311301 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.311300 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="435b529f-0961-4e4f-92da-3db61135f8ab" containerName="cleanup" Jul 21 21:00:13.311428 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.311362 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8862c886-7e33-4048-9061-6decc23738be" containerName="cleanup" Jul 21 21:00:13.311428 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.311369 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="435b529f-0961-4e4f-92da-3db61135f8ab" containerName="cleanup" Jul 21 21:00:13.314505 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.314484 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.316950 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.316923 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-280285-serving-cert\"" Jul 21 21:00:13.327611 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.327586 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d"] Jul 21 21:00:13.381530 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.381497 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966c7\" (UniqueName: \"kubernetes.io/projected/0610435f-c1d8-4b93-af83-1618fb875c77-kube-api-access-966c7\") pod \"maas-api-e2e-api-b-280285-7d684f646c-cs72d\" (UID: \"0610435f-c1d8-4b93-af83-1618fb875c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.381690 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.381558 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0610435f-c1d8-4b93-af83-1618fb875c77-maas-api-tls\") pod \"maas-api-e2e-api-b-280285-7d684f646c-cs72d\" (UID: \"0610435f-c1d8-4b93-af83-1618fb875c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.381690 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.381604 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0610435f-c1d8-4b93-af83-1618fb875c77-openshift-service-ca\") pod \"maas-api-e2e-api-b-280285-7d684f646c-cs72d\" (UID: \"0610435f-c1d8-4b93-af83-1618fb875c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.482178 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.482142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0610435f-c1d8-4b93-af83-1618fb875c77-maas-api-tls\") pod \"maas-api-e2e-api-b-280285-7d684f646c-cs72d\" (UID: \"0610435f-c1d8-4b93-af83-1618fb875c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.482178 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.482184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0610435f-c1d8-4b93-af83-1618fb875c77-openshift-service-ca\") pod \"maas-api-e2e-api-b-280285-7d684f646c-cs72d\" (UID: \"0610435f-c1d8-4b93-af83-1618fb875c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.482458 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.482249 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966c7\" (UniqueName: \"kubernetes.io/projected/0610435f-c1d8-4b93-af83-1618fb875c77-kube-api-access-966c7\") pod \"maas-api-e2e-api-b-280285-7d684f646c-cs72d\" (UID: \"0610435f-c1d8-4b93-af83-1618fb875c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.482930 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.482909 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0610435f-c1d8-4b93-af83-1618fb875c77-openshift-service-ca\") pod \"maas-api-e2e-api-b-280285-7d684f646c-cs72d\" (UID: \"0610435f-c1d8-4b93-af83-1618fb875c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.484654 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.484630 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0610435f-c1d8-4b93-af83-1618fb875c77-maas-api-tls\") pod \"maas-api-e2e-api-b-280285-7d684f646c-cs72d\" (UID: \"0610435f-c1d8-4b93-af83-1618fb875c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.492392 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.492367 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-966c7\" (UniqueName: \"kubernetes.io/projected/0610435f-c1d8-4b93-af83-1618fb875c77-kube-api-access-966c7\") pod \"maas-api-e2e-api-b-280285-7d684f646c-cs72d\" (UID: \"0610435f-c1d8-4b93-af83-1618fb875c77\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.611587 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.611486 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:00:13.628699 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.628653 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:13.782499 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:13.782472 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d"] Jul 21 21:00:13.784039 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:00:13.784008 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0610435f_c1d8_4b93_af83_1618fb875c77.slice/crio-b8a6faf5d09893c209ab306d530472e5f131455fcb77da8386cf16ecd859f5ff WatchSource:0}: Error finding container b8a6faf5d09893c209ab306d530472e5f131455fcb77da8386cf16ecd859f5ff: Status 404 returned error can't find the container with id b8a6faf5d09893c209ab306d530472e5f131455fcb77da8386cf16ecd859f5ff Jul 21 21:00:14.191014 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:14.190970 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" event={"ID":"0610435f-c1d8-4b93-af83-1618fb875c77","Type":"ContainerStarted","Data":"b8a6faf5d09893c209ab306d530472e5f131455fcb77da8386cf16ecd859f5ff"} Jul 21 21:00:15.197310 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:15.197278 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" event={"ID":"0610435f-c1d8-4b93-af83-1618fb875c77","Type":"ContainerStarted","Data":"9ef198a273af6ffcdd5063522ac07cfead1560645389c7e9032ca245ea0417e4"} Jul 21 21:00:15.197731 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:15.197418 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:15.218064 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:15.218013 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" podStartSLOduration=1.803489938 podStartE2EDuration="2.217999301s" totalImagesPullingTime="414.509363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:15.215092954 +0000 UTC m=+2769.895219391" watchObservedRunningTime="2026-07-21 21:00:15.217999301 +0000 UTC m=+2769.898125738" Jul 21 21:00:21.208287 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:21.208258 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-280285-7d684f646c-cs72d" Jul 21 21:00:32.911701 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:32.911605 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:00:45.596114 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:45.596076 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:00:59.826507 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:00:59.826472 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:01:21.209090 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:21.209056 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:01:25.640669 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.640639 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v"] Jul 21 21:01:25.644292 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.644275 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.649400 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.649376 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1ad4ca\"/\"auth-test-model-1ad4ca-kserve-self-signed-certs\"" Jul 21 21:01:25.649400 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.649394 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1ad4ca\"/\"default-dockercfg-pvtlr\"" Jul 21 21:01:25.649586 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.649399 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1ad4ca\"/\"kube-root-ca.crt\"" Jul 21 21:01:25.649586 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.649413 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1ad4ca\"/\"openshift-service-ca.crt\"" Jul 21 21:01:25.736652 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.736616 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v"] Jul 21 21:01:25.751046 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.751012 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-home\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.751175 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.751150 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-dshm\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.751216 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.751183 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-model-cache\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.751252 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.751218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a09436e-44c8-4905-b21a-01969d6133b3-tls-certs\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.751291 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.751248 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-tmp-dir\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.751328 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.751286 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw4nb\" (UniqueName: \"kubernetes.io/projected/3a09436e-44c8-4905-b21a-01969d6133b3-kube-api-access-mw4nb\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.852495 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.852459 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-dshm\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.852495 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.852496 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-model-cache\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.852739 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.852526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a09436e-44c8-4905-b21a-01969d6133b3-tls-certs\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.852739 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.852552 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-tmp-dir\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.852739 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.852586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw4nb\" (UniqueName: \"kubernetes.io/projected/3a09436e-44c8-4905-b21a-01969d6133b3-kube-api-access-mw4nb\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.852739 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.852645 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-home\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.853018 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.852997 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-model-cache\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.853139 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.853031 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-tmp-dir\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.853139 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.853054 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-home\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.854799 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.854779 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-dshm\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.854986 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.854969 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a09436e-44c8-4905-b21a-01969d6133b3-tls-certs\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.862310 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.862283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw4nb\" (UniqueName: \"kubernetes.io/projected/3a09436e-44c8-4905-b21a-01969d6133b3-kube-api-access-mw4nb\") pod \"auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:25.953845 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:25.953763 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:26.098149 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:26.098123 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v"] Jul 21 21:01:26.100129 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:01:26.100103 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a09436e_44c8_4905_b21a_01969d6133b3.slice/crio-75ec8d16d0a89b99a16e39303d3de783a4fcc72becad740545c4606aca389598 WatchSource:0}: Error finding container 75ec8d16d0a89b99a16e39303d3de783a4fcc72becad740545c4606aca389598: Status 404 returned error can't find the container with id 75ec8d16d0a89b99a16e39303d3de783a4fcc72becad740545c4606aca389598 Jul 21 21:01:26.612580 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:26.612548 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" event={"ID":"3a09436e-44c8-4905-b21a-01969d6133b3","Type":"ContainerStarted","Data":"9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd"} Jul 21 21:01:26.612580 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:26.612594 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" event={"ID":"3a09436e-44c8-4905-b21a-01969d6133b3","Type":"ContainerStarted","Data":"75ec8d16d0a89b99a16e39303d3de783a4fcc72becad740545c4606aca389598"} Jul 21 21:01:26.636454 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:26.636393 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" podStartSLOduration=1.636376436 podStartE2EDuration="1.636376436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:26.633632434 +0000 UTC m=+2841.313758873" watchObservedRunningTime="2026-07-21 21:01:26.636376436 +0000 UTC m=+2841.316502930" Jul 21 21:01:32.099661 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:32.099626 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:01:35.954067 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:35.954025 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:35.954067 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:35.954079 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:35.966986 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:35.966924 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:36.681415 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:36.681388 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:01:50.991051 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:50.991016 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w"] Jul 21 21:01:50.994683 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:50.994665 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:50.998122 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:50.998100 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"kube-root-ca.crt\"" Jul 21 21:01:50.998230 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:50.998105 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"default-dockercfg-6nmwn\"" Jul 21 21:01:50.998230 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:50.998107 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"auth-test-model-9b964f-kserve-self-signed-certs\"" Jul 21 21:01:50.998342 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:50.998242 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"openshift-service-ca.crt\"" Jul 21 21:01:51.006586 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.006566 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w"] Jul 21 21:01:51.088571 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.088536 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tls-certs\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.088736 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.088583 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-dshm\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.088736 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.088671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-model-cache\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.088736 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.088713 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gdq\" (UniqueName: \"kubernetes.io/projected/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-kube-api-access-m4gdq\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.088847 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.088736 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tmp-dir\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.088847 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.088820 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-home\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.189721 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.189675 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-model-cache\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.189721 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.189727 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gdq\" (UniqueName: \"kubernetes.io/projected/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-kube-api-access-m4gdq\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.189994 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.189836 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tmp-dir\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.189994 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.189890 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-home\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.189994 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.189973 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tls-certs\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.190168 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.190026 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-dshm\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.190168 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.190119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-model-cache\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.190281 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.190247 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tmp-dir\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.190323 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.190302 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-home\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.192332 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.192312 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-dshm\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.192497 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.192480 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tls-certs\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.198310 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.198289 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gdq\" (UniqueName: \"kubernetes.io/projected/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-kube-api-access-m4gdq\") pod \"auth-test-model-9b964f-kserve-54d7b66747-9r77w\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.305331 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.305250 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:01:51.444067 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.444042 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w"] Jul 21 21:01:51.445975 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:01:51.445922 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3674e6c7_e6cf_447e_8e37_cd4b2f86d714.slice/crio-102411d4243d126930d372ac4feb93573911331ba220166837a652fc88b38b54 WatchSource:0}: Error finding container 102411d4243d126930d372ac4feb93573911331ba220166837a652fc88b38b54: Status 404 returned error can't find the container with id 102411d4243d126930d372ac4feb93573911331ba220166837a652fc88b38b54 Jul 21 21:01:51.758063 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.758017 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" event={"ID":"3674e6c7-e6cf-447e-8e37-cd4b2f86d714","Type":"ContainerStarted","Data":"3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c"} Jul 21 21:01:51.758063 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.758067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" event={"ID":"3674e6c7-e6cf-447e-8e37-cd4b2f86d714","Type":"ContainerStarted","Data":"102411d4243d126930d372ac4feb93573911331ba220166837a652fc88b38b54"} Jul 21 21:01:51.777621 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:51.777572 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" podStartSLOduration=1.777556942 podStartE2EDuration="1.777556942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:51.77548553 +0000 UTC m=+2866.455611983" watchObservedRunningTime="2026-07-21 21:01:51.777556942 +0000 UTC m=+2866.457683379" Jul 21 21:01:55.087350 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:01:55.087318 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:02:01.306318 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:02:01.306225 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:02:01.306318 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:02:01.306278 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:02:01.318754 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:02:01.318730 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:02:01.825598 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:02:01.825569 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:02:25.585003 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:02:25.584965 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:02:44.381173 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:02:44.381137 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:03:01.590528 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:03:01.590491 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:03:22.186315 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:03:22.186278 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:03:43.888160 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:03:43.888085 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:04:03.089726 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:03.089688 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:04:06.007619 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:06.007589 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 21:04:06.019299 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:06.019278 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nzmdl_fa25280b-6271-4fb4-9961-a9ecdb0ba5ab/ovn-acl-logging/0.log" Jul 21 21:04:23.708762 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:23.708710 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:04:28.621624 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:28.621592 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v"] Jul 21 21:04:29.635456 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:29.635425 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w"] Jul 21 21:04:43.641453 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.641409 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" podUID="3a09436e-44c8-4905-b21a-01969d6133b3" containerName="main" containerID="cri-o://9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd" gracePeriod=45 Jul 21 21:04:43.909667 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.909641 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:04:43.988039 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988011 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw4nb\" (UniqueName: \"kubernetes.io/projected/3a09436e-44c8-4905-b21a-01969d6133b3-kube-api-access-mw4nb\") pod \"3a09436e-44c8-4905-b21a-01969d6133b3\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " Jul 21 21:04:43.988231 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988072 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-home\") pod \"3a09436e-44c8-4905-b21a-01969d6133b3\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " Jul 21 21:04:43.988231 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988106 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a09436e-44c8-4905-b21a-01969d6133b3-tls-certs\") pod \"3a09436e-44c8-4905-b21a-01969d6133b3\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " Jul 21 21:04:43.988231 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988142 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-model-cache\") pod \"3a09436e-44c8-4905-b21a-01969d6133b3\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " Jul 21 21:04:43.988231 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988191 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-dshm\") pod \"3a09436e-44c8-4905-b21a-01969d6133b3\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " Jul 21 21:04:43.988231 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988215 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-tmp-dir\") pod \"3a09436e-44c8-4905-b21a-01969d6133b3\" (UID: \"3a09436e-44c8-4905-b21a-01969d6133b3\") " Jul 21 21:04:43.988486 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988343 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-home" (OuterVolumeSpecName: "home") pod "3a09436e-44c8-4905-b21a-01969d6133b3" (UID: "3a09436e-44c8-4905-b21a-01969d6133b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:04:43.988548 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988510 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a09436e-44c8-4905-b21a-01969d6133b3" (UID: "3a09436e-44c8-4905-b21a-01969d6133b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:04:43.988601 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988536 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a09436e-44c8-4905-b21a-01969d6133b3" (UID: "3a09436e-44c8-4905-b21a-01969d6133b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:04:43.988678 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988659 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-tmp-dir\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:43.988721 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988682 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-home\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:43.988721 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.988696 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-model-cache\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:43.990355 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.990327 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-dshm" (OuterVolumeSpecName: "dshm") pod "3a09436e-44c8-4905-b21a-01969d6133b3" (UID: "3a09436e-44c8-4905-b21a-01969d6133b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:04:43.990459 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.990405 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a09436e-44c8-4905-b21a-01969d6133b3-kube-api-access-mw4nb" (OuterVolumeSpecName: "kube-api-access-mw4nb") pod "3a09436e-44c8-4905-b21a-01969d6133b3" (UID: "3a09436e-44c8-4905-b21a-01969d6133b3"). InnerVolumeSpecName "kube-api-access-mw4nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:04:43.990459 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:43.990406 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a09436e-44c8-4905-b21a-01969d6133b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a09436e-44c8-4905-b21a-01969d6133b3" (UID: "3a09436e-44c8-4905-b21a-01969d6133b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:04:44.089566 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.089532 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a09436e-44c8-4905-b21a-01969d6133b3-tls-certs\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:44.089566 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.089559 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a09436e-44c8-4905-b21a-01969d6133b3-dshm\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:44.089566 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.089571 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw4nb\" (UniqueName: \"kubernetes.io/projected/3a09436e-44c8-4905-b21a-01969d6133b3-kube-api-access-mw4nb\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:44.655473 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.655412 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" podUID="3674e6c7-e6cf-447e-8e37-cd4b2f86d714" containerName="main" containerID="cri-o://3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c" gracePeriod=45 Jul 21 21:04:44.760071 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.760034 2567 generic.go:356] "Generic (PLEG): container finished" podID="3a09436e-44c8-4905-b21a-01969d6133b3" containerID="9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd" exitCode=0 Jul 21 21:04:44.760229 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.760101 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" Jul 21 21:04:44.760229 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.760124 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" event={"ID":"3a09436e-44c8-4905-b21a-01969d6133b3","Type":"ContainerDied","Data":"9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd"} Jul 21 21:04:44.760229 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.760169 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v" event={"ID":"3a09436e-44c8-4905-b21a-01969d6133b3","Type":"ContainerDied","Data":"75ec8d16d0a89b99a16e39303d3de783a4fcc72becad740545c4606aca389598"} Jul 21 21:04:44.760229 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.760193 2567 scope.go:117] "RemoveContainer" containerID="9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd" Jul 21 21:04:44.769310 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.769289 2567 scope.go:117] "RemoveContainer" containerID="9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd" Jul 21 21:04:44.769606 ip-10-0-141-132 kubenswrapper[2567]: E0721 21:04:44.769581 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd\": container with ID starting with 9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd not found: ID does not exist" containerID="9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd" Jul 21 21:04:44.769659 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.769621 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd"} err="failed to get container status \"9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd\": rpc error: code = NotFound desc = could not find container \"9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd\": container with ID starting with 9256ef7b15101d4d94a14bb9ac6ed453eb5358ac5122b32dca7a981ba869d0dd not found: ID does not exist" Jul 21 21:04:44.803639 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.803601 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v"] Jul 21 21:04:44.808167 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.808140 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/auth-test-model-1ad4ca-kserve-56b99df7c9-8f28v"] Jul 21 21:04:44.914577 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.914509 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:04:44.998052 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998018 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-home\") pod \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " Jul 21 21:04:44.998255 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998092 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tls-certs\") pod \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " Jul 21 21:04:44.998255 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998110 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4gdq\" (UniqueName: \"kubernetes.io/projected/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-kube-api-access-m4gdq\") pod \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " Jul 21 21:04:44.998255 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998169 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-dshm\") pod \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " Jul 21 21:04:44.998423 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998302 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-model-cache\") pod \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " Jul 21 21:04:44.998423 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998311 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-home" (OuterVolumeSpecName: "home") pod "3674e6c7-e6cf-447e-8e37-cd4b2f86d714" (UID: "3674e6c7-e6cf-447e-8e37-cd4b2f86d714"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:04:44.998423 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998357 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tmp-dir\") pod \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\" (UID: \"3674e6c7-e6cf-447e-8e37-cd4b2f86d714\") " Jul 21 21:04:44.998581 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998547 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-model-cache" (OuterVolumeSpecName: "model-cache") pod "3674e6c7-e6cf-447e-8e37-cd4b2f86d714" (UID: "3674e6c7-e6cf-447e-8e37-cd4b2f86d714"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:04:44.998581 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998561 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3674e6c7-e6cf-447e-8e37-cd4b2f86d714" (UID: "3674e6c7-e6cf-447e-8e37-cd4b2f86d714"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:04:44.998720 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998698 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-home\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:44.998720 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998722 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-model-cache\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:44.998923 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:44.998736 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tmp-dir\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:45.000462 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.000438 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-dshm" (OuterVolumeSpecName: "dshm") pod "3674e6c7-e6cf-447e-8e37-cd4b2f86d714" (UID: "3674e6c7-e6cf-447e-8e37-cd4b2f86d714"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:04:45.000552 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.000479 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3674e6c7-e6cf-447e-8e37-cd4b2f86d714" (UID: "3674e6c7-e6cf-447e-8e37-cd4b2f86d714"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:04:45.000596 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.000557 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-kube-api-access-m4gdq" (OuterVolumeSpecName: "kube-api-access-m4gdq") pod "3674e6c7-e6cf-447e-8e37-cd4b2f86d714" (UID: "3674e6c7-e6cf-447e-8e37-cd4b2f86d714"). InnerVolumeSpecName "kube-api-access-m4gdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:04:45.099408 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.099375 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-tls-certs\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:45.099408 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.099407 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4gdq\" (UniqueName: \"kubernetes.io/projected/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-kube-api-access-m4gdq\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:45.099607 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.099419 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3674e6c7-e6cf-447e-8e37-cd4b2f86d714-dshm\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:04:45.190050 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.189954 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:04:45.767544 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.767507 2567 generic.go:356] "Generic (PLEG): container finished" podID="3674e6c7-e6cf-447e-8e37-cd4b2f86d714" containerID="3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c" exitCode=0 Jul 21 21:04:45.768068 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.767552 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" Jul 21 21:04:45.768068 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.767591 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" event={"ID":"3674e6c7-e6cf-447e-8e37-cd4b2f86d714","Type":"ContainerDied","Data":"3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c"} Jul 21 21:04:45.768068 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.767631 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w" event={"ID":"3674e6c7-e6cf-447e-8e37-cd4b2f86d714","Type":"ContainerDied","Data":"102411d4243d126930d372ac4feb93573911331ba220166837a652fc88b38b54"} Jul 21 21:04:45.768068 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.767648 2567 scope.go:117] "RemoveContainer" containerID="3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c" Jul 21 21:04:45.777847 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.777807 2567 scope.go:117] "RemoveContainer" containerID="3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c" Jul 21 21:04:45.778119 ip-10-0-141-132 kubenswrapper[2567]: E0721 21:04:45.778102 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c\": container with ID starting with 3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c not found: ID does not exist" containerID="3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c" Jul 21 21:04:45.778175 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.778132 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c"} err="failed to get container status \"3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c\": rpc error: code = NotFound desc = could not find container \"3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c\": container with ID starting with 3b60b0fb662ff74c37daa1a8de7fbfa41386941dd67e40eed700154455ff893c not found: ID does not exist" Jul 21 21:04:45.792881 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.792853 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w"] Jul 21 21:04:45.806205 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.806175 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a09436e-44c8-4905-b21a-01969d6133b3" path="/var/lib/kubelet/pods/3a09436e-44c8-4905-b21a-01969d6133b3/volumes" Jul 21 21:04:45.806461 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:45.806447 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/auth-test-model-9b964f-kserve-54d7b66747-9r77w"] Jul 21 21:04:47.804963 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:47.804910 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3674e6c7-e6cf-447e-8e37-cd4b2f86d714" path="/var/lib/kubelet/pods/3674e6c7-e6cf-447e-8e37-cd4b2f86d714/volumes" Jul 21 21:04:59.806797 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:04:59.806721 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:05:08.914228 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.914189 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd"] Jul 21 21:05:08.914635 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.914621 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a09436e-44c8-4905-b21a-01969d6133b3" containerName="main" Jul 21 21:05:08.914675 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.914636 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a09436e-44c8-4905-b21a-01969d6133b3" containerName="main" Jul 21 21:05:08.914675 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.914645 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3674e6c7-e6cf-447e-8e37-cd4b2f86d714" containerName="main" Jul 21 21:05:08.914675 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.914650 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3674e6c7-e6cf-447e-8e37-cd4b2f86d714" containerName="main" Jul 21 21:05:08.914772 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.914743 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a09436e-44c8-4905-b21a-01969d6133b3" containerName="main" Jul 21 21:05:08.914772 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.914756 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3674e6c7-e6cf-447e-8e37-cd4b2f86d714" containerName="main" Jul 21 21:05:08.919214 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.919196 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:08.922804 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.922779 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"kube-root-ca.crt\"" Jul 21 21:05:08.922967 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.922782 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"default-dockercfg-6nmwn\"" Jul 21 21:05:08.923047 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.923001 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"openshift-service-ca.crt\"" Jul 21 21:05:08.923047 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.923016 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"sub-test-model-9b964f-kserve-self-signed-certs\"" Jul 21 21:05:08.932232 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:08.932204 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd"] Jul 21 21:05:09.028589 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.028555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-home\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.028749 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.028600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ec5d87-9cba-4aec-91bb-585047a659a5-tls-certs\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.028749 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.028635 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-model-cache\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.028749 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.028657 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-tmp-dir\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.028749 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.028694 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbf6\" (UniqueName: \"kubernetes.io/projected/81ec5d87-9cba-4aec-91bb-585047a659a5-kube-api-access-dlbf6\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.028919 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.028766 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-dshm\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.130300 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.130262 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-dshm\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.130500 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.130322 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-home\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.130500 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.130358 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ec5d87-9cba-4aec-91bb-585047a659a5-tls-certs\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.130500 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.130388 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-model-cache\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.130500 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.130411 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-tmp-dir\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.130500 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.130456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbf6\" (UniqueName: \"kubernetes.io/projected/81ec5d87-9cba-4aec-91bb-585047a659a5-kube-api-access-dlbf6\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.130790 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.130771 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-home\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.130845 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.130814 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-tmp-dir\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.130893 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.130855 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-model-cache\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.132577 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.132553 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-dshm\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.132820 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.132804 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ec5d87-9cba-4aec-91bb-585047a659a5-tls-certs\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.140859 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.140835 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbf6\" (UniqueName: \"kubernetes.io/projected/81ec5d87-9cba-4aec-91bb-585047a659a5-kube-api-access-dlbf6\") pod \"sub-test-model-9b964f-kserve-64897b4795-kn8gd\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.228777 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.228684 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:09.371094 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.371059 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd"] Jul 21 21:05:09.372270 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:05:09.372245 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81ec5d87_9cba_4aec_91bb_585047a659a5.slice/crio-06337b23689b796fdbab0d568d4d1fe8a2ec2e214f869a53c80aff95e60c7253 WatchSource:0}: Error finding container 06337b23689b796fdbab0d568d4d1fe8a2ec2e214f869a53c80aff95e60c7253: Status 404 returned error can't find the container with id 06337b23689b796fdbab0d568d4d1fe8a2ec2e214f869a53c80aff95e60c7253 Jul 21 21:05:09.908341 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.908303 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" event={"ID":"81ec5d87-9cba-4aec-91bb-585047a659a5","Type":"ContainerStarted","Data":"931fec13ed96cc93170f76e06e638fec4335096d6956a303940d604fb7aa7202"} Jul 21 21:05:09.908617 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.908352 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" event={"ID":"81ec5d87-9cba-4aec-91bb-585047a659a5","Type":"ContainerStarted","Data":"06337b23689b796fdbab0d568d4d1fe8a2ec2e214f869a53c80aff95e60c7253"} Jul 21 21:05:09.932566 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:09.932503 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" podStartSLOduration=1.9324891709999998 podStartE2EDuration="1.932489171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:05:09.931822472 +0000 UTC m=+3064.611948927" watchObservedRunningTime="2026-07-21 21:05:09.932489171 +0000 UTC m=+3064.612615607" Jul 21 21:05:13.195814 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:13.195778 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:05:19.229865 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:19.229822 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:19.230283 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:19.229880 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:19.242141 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:19.242109 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:19.982087 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:19.982056 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:05:30.913214 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:30.913175 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:05:53.504250 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:05:53.504212 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:06:01.126494 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:01.126455 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd"] Jul 21 21:06:12.699434 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.699401 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw"] Jul 21 21:06:12.703101 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.703084 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.705720 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.705697 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1ad4ca\"/\"openshift-service-ca.crt\"" Jul 21 21:06:12.706311 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.706284 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1ad4ca\"/\"kube-root-ca.crt\"" Jul 21 21:06:12.706444 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.706425 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1ad4ca\"/\"default-dockercfg-pvtlr\"" Jul 21 21:06:12.706537 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.706518 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1ad4ca\"/\"rate-test-model-1ad4ca-kserve-self-signed-certs\"" Jul 21 21:06:12.712416 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.712396 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw"] Jul 21 21:06:12.826626 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.826593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5mld\" (UniqueName: \"kubernetes.io/projected/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-kube-api-access-p5mld\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.826823 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.826728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-dshm\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.826823 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.826774 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-home\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.826823 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.826799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tmp-dir\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.827034 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.826823 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-model-cache\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.827034 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.826883 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tls-certs\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.927987 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.927919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-home\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.928187 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.928000 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tmp-dir\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.928187 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.928022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-model-cache\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.928187 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.928045 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tls-certs\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.928360 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.928183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mld\" (UniqueName: \"kubernetes.io/projected/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-kube-api-access-p5mld\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.928360 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.928329 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-dshm\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.928509 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.928415 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-home\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.928509 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.928446 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tmp-dir\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.928509 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.928466 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-model-cache\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.930451 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.930417 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-dshm\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.930682 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.930667 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tls-certs\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:12.936169 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:12.936144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mld\" (UniqueName: \"kubernetes.io/projected/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-kube-api-access-p5mld\") pod \"rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:13.013171 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:13.013075 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:13.153532 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:13.153486 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw"] Jul 21 21:06:13.156580 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:06:13.156547 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3343a8ed_e9ec_4e00_82b8_f25a85ee718e.slice/crio-3f5af60605cc473f3afca4d0502e9c416d548218754338337c02cf022c4beaa6 WatchSource:0}: Error finding container 3f5af60605cc473f3afca4d0502e9c416d548218754338337c02cf022c4beaa6: Status 404 returned error can't find the container with id 3f5af60605cc473f3afca4d0502e9c416d548218754338337c02cf022c4beaa6 Jul 21 21:06:13.287481 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:13.287377 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" event={"ID":"3343a8ed-e9ec-4e00-82b8-f25a85ee718e","Type":"ContainerStarted","Data":"ff26cbe81a6a2a8ea8fb867155e9680372576b3a7fbb67b747cadd69fe4661c6"} Jul 21 21:06:13.287481 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:13.287424 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" event={"ID":"3343a8ed-e9ec-4e00-82b8-f25a85ee718e","Type":"ContainerStarted","Data":"3f5af60605cc473f3afca4d0502e9c416d548218754338337c02cf022c4beaa6"} Jul 21 21:06:13.306387 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:13.306328 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" podStartSLOduration=1.306313695 podStartE2EDuration="1.306313695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:13.304519852 +0000 UTC m=+3127.984646301" watchObservedRunningTime="2026-07-21 21:06:13.306313695 +0000 UTC m=+3127.986440132" Jul 21 21:06:16.147530 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.147449 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" podUID="81ec5d87-9cba-4aec-91bb-585047a659a5" containerName="main" containerID="cri-o://931fec13ed96cc93170f76e06e638fec4335096d6956a303940d604fb7aa7202" gracePeriod=45 Jul 21 21:06:16.287183 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.287146 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:06:16.306346 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.306310 2567 generic.go:356] "Generic (PLEG): container finished" podID="81ec5d87-9cba-4aec-91bb-585047a659a5" containerID="931fec13ed96cc93170f76e06e638fec4335096d6956a303940d604fb7aa7202" exitCode=0 Jul 21 21:06:16.306551 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.306473 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" event={"ID":"81ec5d87-9cba-4aec-91bb-585047a659a5","Type":"ContainerDied","Data":"931fec13ed96cc93170f76e06e638fec4335096d6956a303940d604fb7aa7202"} Jul 21 21:06:16.415330 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.415256 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:06:16.462346 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.462308 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-tmp-dir\") pod \"81ec5d87-9cba-4aec-91bb-585047a659a5\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " Jul 21 21:06:16.462548 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.462363 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ec5d87-9cba-4aec-91bb-585047a659a5-tls-certs\") pod \"81ec5d87-9cba-4aec-91bb-585047a659a5\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " Jul 21 21:06:16.462548 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.462415 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlbf6\" (UniqueName: \"kubernetes.io/projected/81ec5d87-9cba-4aec-91bb-585047a659a5-kube-api-access-dlbf6\") pod \"81ec5d87-9cba-4aec-91bb-585047a659a5\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " Jul 21 21:06:16.462548 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.462446 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-model-cache\") pod \"81ec5d87-9cba-4aec-91bb-585047a659a5\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " Jul 21 21:06:16.462548 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.462491 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-home\") pod \"81ec5d87-9cba-4aec-91bb-585047a659a5\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " Jul 21 21:06:16.462774 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.462563 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-dshm\") pod \"81ec5d87-9cba-4aec-91bb-585047a659a5\" (UID: \"81ec5d87-9cba-4aec-91bb-585047a659a5\") " Jul 21 21:06:16.462774 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.462597 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81ec5d87-9cba-4aec-91bb-585047a659a5" (UID: "81ec5d87-9cba-4aec-91bb-585047a659a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:06:16.462852 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.462767 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-home" (OuterVolumeSpecName: "home") pod "81ec5d87-9cba-4aec-91bb-585047a659a5" (UID: "81ec5d87-9cba-4aec-91bb-585047a659a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:06:16.462954 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.462919 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "81ec5d87-9cba-4aec-91bb-585047a659a5" (UID: "81ec5d87-9cba-4aec-91bb-585047a659a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:06:16.463308 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.463281 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-tmp-dir\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:06:16.463308 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.463311 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-model-cache\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:06:16.463499 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.463324 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-home\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:06:16.464754 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.464725 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-dshm" (OuterVolumeSpecName: "dshm") pod "81ec5d87-9cba-4aec-91bb-585047a659a5" (UID: "81ec5d87-9cba-4aec-91bb-585047a659a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:06:16.465076 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.465054 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ec5d87-9cba-4aec-91bb-585047a659a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81ec5d87-9cba-4aec-91bb-585047a659a5" (UID: "81ec5d87-9cba-4aec-91bb-585047a659a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:06:16.465175 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.465059 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ec5d87-9cba-4aec-91bb-585047a659a5-kube-api-access-dlbf6" (OuterVolumeSpecName: "kube-api-access-dlbf6") pod "81ec5d87-9cba-4aec-91bb-585047a659a5" (UID: "81ec5d87-9cba-4aec-91bb-585047a659a5"). InnerVolumeSpecName "kube-api-access-dlbf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:06:16.564455 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.564416 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81ec5d87-9cba-4aec-91bb-585047a659a5-dshm\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:06:16.564455 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.564457 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81ec5d87-9cba-4aec-91bb-585047a659a5-tls-certs\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:06:16.564711 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:16.564473 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbf6\" (UniqueName: \"kubernetes.io/projected/81ec5d87-9cba-4aec-91bb-585047a659a5-kube-api-access-dlbf6\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:06:17.313142 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:17.313100 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" Jul 21 21:06:17.313587 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:17.313107 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd" event={"ID":"81ec5d87-9cba-4aec-91bb-585047a659a5","Type":"ContainerDied","Data":"06337b23689b796fdbab0d568d4d1fe8a2ec2e214f869a53c80aff95e60c7253"} Jul 21 21:06:17.313587 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:17.313223 2567 scope.go:117] "RemoveContainer" containerID="931fec13ed96cc93170f76e06e638fec4335096d6956a303940d604fb7aa7202" Jul 21 21:06:17.337305 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:17.337269 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd"] Jul 21 21:06:17.342141 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:17.342102 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/sub-test-model-9b964f-kserve-64897b4795-kn8gd"] Jul 21 21:06:17.806033 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:17.806004 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81ec5d87-9cba-4aec-91bb-585047a659a5" path="/var/lib/kubelet/pods/81ec5d87-9cba-4aec-91bb-585047a659a5/volumes" Jul 21 21:06:23.014008 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:23.013972 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:23.014406 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:23.014020 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:23.026622 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:23.026599 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:23.360399 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:23.360323 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:06:31.196632 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.196547 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7"] Jul 21 21:06:31.197057 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.197011 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ec5d87-9cba-4aec-91bb-585047a659a5" containerName="main" Jul 21 21:06:31.197057 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.197023 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ec5d87-9cba-4aec-91bb-585047a659a5" containerName="main" Jul 21 21:06:31.197141 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.197096 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ec5d87-9cba-4aec-91bb-585047a659a5" containerName="main" Jul 21 21:06:31.200557 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.200537 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.203987 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.203959 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"openshift-service-ca.crt\"" Jul 21 21:06:31.204121 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.203995 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"default-dockercfg-6nmwn\"" Jul 21 21:06:31.204121 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.203970 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"kube-root-ca.crt\"" Jul 21 21:06:31.204121 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.203974 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9b964f\"/\"rate-test-model-9b964f-kserve-self-signed-certs\"" Jul 21 21:06:31.209778 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.209754 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7"] Jul 21 21:06:31.300390 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.300356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-dshm\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.300390 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.300395 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wvpr\" (UniqueName: \"kubernetes.io/projected/12cd6373-e6c0-480e-a3c0-57d58db46d64-kube-api-access-4wvpr\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.300608 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.300497 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-model-cache\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.300608 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.300573 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd6373-e6c0-480e-a3c0-57d58db46d64-tls-certs\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.300608 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.300600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-home\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.300711 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.300615 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-tmp-dir\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.401732 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.401701 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-dshm\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.401732 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.401734 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wvpr\" (UniqueName: \"kubernetes.io/projected/12cd6373-e6c0-480e-a3c0-57d58db46d64-kube-api-access-4wvpr\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.402019 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.401872 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-model-cache\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.402019 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.401972 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd6373-e6c0-480e-a3c0-57d58db46d64-tls-certs\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.402019 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.402005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-home\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.402189 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.402027 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-tmp-dir\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.402313 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.402293 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-model-cache\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.402369 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.402323 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-tmp-dir\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.402422 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.402404 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-home\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.404228 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.404207 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-dshm\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.404430 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.404414 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd6373-e6c0-480e-a3c0-57d58db46d64-tls-certs\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.412486 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.412458 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wvpr\" (UniqueName: \"kubernetes.io/projected/12cd6373-e6c0-480e-a3c0-57d58db46d64-kube-api-access-4wvpr\") pod \"rate-test-model-9b964f-kserve-5589f6b88c-tjpt7\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.511686 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.511600 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:31.650012 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:31.649987 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7"] Jul 21 21:06:31.651311 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:06:31.651284 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12cd6373_e6c0_480e_a3c0_57d58db46d64.slice/crio-2039d3f48c505b5197600d4c1b62812d2bfc1eb955a1fdd396b589e848cea91a WatchSource:0}: Error finding container 2039d3f48c505b5197600d4c1b62812d2bfc1eb955a1fdd396b589e848cea91a: Status 404 returned error can't find the container with id 2039d3f48c505b5197600d4c1b62812d2bfc1eb955a1fdd396b589e848cea91a Jul 21 21:06:32.398884 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:32.398843 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" event={"ID":"12cd6373-e6c0-480e-a3c0-57d58db46d64","Type":"ContainerStarted","Data":"6fa9eb05a4b74611ee490db37b5f7c1e37fb0a7e2e9dd7a77dc2865fcf0a1a51"} Jul 21 21:06:32.398884 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:32.398891 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" event={"ID":"12cd6373-e6c0-480e-a3c0-57d58db46d64","Type":"ContainerStarted","Data":"2039d3f48c505b5197600d4c1b62812d2bfc1eb955a1fdd396b589e848cea91a"} Jul 21 21:06:32.422470 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:32.422420 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" podStartSLOduration=1.422404434 podStartE2EDuration="1.422404434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:32.420447366 +0000 UTC m=+3147.100573814" watchObservedRunningTime="2026-07-21 21:06:32.422404434 +0000 UTC m=+3147.102530902" Jul 21 21:06:35.997400 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:35.997364 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:06:41.512123 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:41.512088 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:41.512123 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:41.512126 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:41.524683 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:41.524660 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:06:42.473093 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:06:42.473064 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:07:12.600083 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:12.600045 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:07:29.488478 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:29.488443 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:07:36.687576 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:36.687540 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw"] Jul 21 21:07:37.680826 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:37.680791 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7"] Jul 21 21:07:38.629928 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:38.629898 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7138-140442618113200"] Jul 21 21:07:38.633605 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:38.633584 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442618113200" Jul 21 21:07:38.636157 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:38.636139 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pr87k\"" Jul 21 21:07:38.640565 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:38.640539 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442618113200"] Jul 21 21:07:38.734003 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:38.733975 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vkt\" (UniqueName: \"kubernetes.io/projected/7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b-kube-api-access-n2vkt\") pod \"test-curl-7138-140442618113200\" (UID: \"7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b\") " pod="opendatahub/test-curl-7138-140442618113200" Jul 21 21:07:38.835021 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:38.834984 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vkt\" (UniqueName: \"kubernetes.io/projected/7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b-kube-api-access-n2vkt\") pod \"test-curl-7138-140442618113200\" (UID: \"7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b\") " pod="opendatahub/test-curl-7138-140442618113200" Jul 21 21:07:38.844045 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:38.844020 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vkt\" (UniqueName: \"kubernetes.io/projected/7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b-kube-api-access-n2vkt\") pod \"test-curl-7138-140442618113200\" (UID: \"7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b\") " pod="opendatahub/test-curl-7138-140442618113200" Jul 21 21:07:38.945314 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:38.945284 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442618113200" Jul 21 21:07:39.079370 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:39.079348 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442618113200"] Jul 21 21:07:39.081493 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:07:39.081468 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7166f3cd_f1e1_49ae_a2fb_0b1f1f64301b.slice/crio-2c8f40612fa54ae12b7980a9dfff45f507934a3de5b8f516e4df5b553af4ea3f WatchSource:0}: Error finding container 2c8f40612fa54ae12b7980a9dfff45f507934a3de5b8f516e4df5b553af4ea3f: Status 404 returned error can't find the container with id 2c8f40612fa54ae12b7980a9dfff45f507934a3de5b8f516e4df5b553af4ea3f Jul 21 21:07:39.082684 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:39.082667 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:07:39.789415 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:39.789389 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442618113200" event={"ID":"7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b","Type":"ContainerStarted","Data":"2c8f40612fa54ae12b7980a9dfff45f507934a3de5b8f516e4df5b553af4ea3f"} Jul 21 21:07:40.797346 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:40.797316 2567 generic.go:356] "Generic (PLEG): container finished" podID="7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b" containerID="0e993d1f26b2c514f1d6b7ee2c4cb85c8546c295b8e5f6a60eb2bd06e89ae6bb" exitCode=0 Jul 21 21:07:40.797346 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:40.797351 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442618113200" event={"ID":"7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b","Type":"ContainerDied","Data":"0e993d1f26b2c514f1d6b7ee2c4cb85c8546c295b8e5f6a60eb2bd06e89ae6bb"} Jul 21 21:07:41.937335 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:41.937310 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442618113200" Jul 21 21:07:41.965709 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:41.965679 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2vkt\" (UniqueName: \"kubernetes.io/projected/7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b-kube-api-access-n2vkt\") pod \"7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b\" (UID: \"7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b\") " Jul 21 21:07:41.967929 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:41.967905 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b-kube-api-access-n2vkt" (OuterVolumeSpecName: "kube-api-access-n2vkt") pod "7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b" (UID: "7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b"). InnerVolumeSpecName "kube-api-access-n2vkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:41.975673 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:41.975646 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7138-140442618113200_7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b/test-curl-7138-140442618113200/0.log" Jul 21 21:07:41.998180 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:41.998152 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7138-140442618113200"] Jul 21 21:07:42.002156 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.002125 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7138-140442618113200"] Jul 21 21:07:42.067156 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.067072 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vkt\" (UniqueName: \"kubernetes.io/projected/7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b-kube-api-access-n2vkt\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:42.118400 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.118369 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7138-140442617865520"] Jul 21 21:07:42.118769 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.118756 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b" containerName="test-curl-7138-140442618113200" Jul 21 21:07:42.118812 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.118770 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b" containerName="test-curl-7138-140442618113200" Jul 21 21:07:42.118857 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.118847 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b" containerName="test-curl-7138-140442618113200" Jul 21 21:07:42.121993 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.121977 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617865520" Jul 21 21:07:42.129332 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.129307 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442617865520"] Jul 21 21:07:42.167781 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.167751 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td6jp\" (UniqueName: \"kubernetes.io/projected/a82629e7-1635-417b-bdec-48638d2b1818-kube-api-access-td6jp\") pod \"test-curl-7138-140442617865520\" (UID: \"a82629e7-1635-417b-bdec-48638d2b1818\") " pod="opendatahub/test-curl-7138-140442617865520" Jul 21 21:07:42.269191 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.269159 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td6jp\" (UniqueName: \"kubernetes.io/projected/a82629e7-1635-417b-bdec-48638d2b1818-kube-api-access-td6jp\") pod \"test-curl-7138-140442617865520\" (UID: \"a82629e7-1635-417b-bdec-48638d2b1818\") " pod="opendatahub/test-curl-7138-140442617865520" Jul 21 21:07:42.283386 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.283356 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td6jp\" (UniqueName: \"kubernetes.io/projected/a82629e7-1635-417b-bdec-48638d2b1818-kube-api-access-td6jp\") pod \"test-curl-7138-140442617865520\" (UID: \"a82629e7-1635-417b-bdec-48638d2b1818\") " pod="opendatahub/test-curl-7138-140442617865520" Jul 21 21:07:42.432688 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.432657 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617865520" Jul 21 21:07:42.775361 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.775334 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442617865520"] Jul 21 21:07:42.776538 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:07:42.776504 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda82629e7_1635_417b_bdec_48638d2b1818.slice/crio-19a28a5467b439238f3ee7ceb17d44575e0ab54ffc5c9f84148701b5eec42a15 WatchSource:0}: Error finding container 19a28a5467b439238f3ee7ceb17d44575e0ab54ffc5c9f84148701b5eec42a15: Status 404 returned error can't find the container with id 19a28a5467b439238f3ee7ceb17d44575e0ab54ffc5c9f84148701b5eec42a15 Jul 21 21:07:42.810255 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.810231 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442618113200" Jul 21 21:07:42.810255 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.810243 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c8f40612fa54ae12b7980a9dfff45f507934a3de5b8f516e4df5b553af4ea3f" Jul 21 21:07:42.811654 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:42.811629 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442617865520" event={"ID":"a82629e7-1635-417b-bdec-48638d2b1818","Type":"ContainerStarted","Data":"19a28a5467b439238f3ee7ceb17d44575e0ab54ffc5c9f84148701b5eec42a15"} Jul 21 21:07:42.821469 ip-10-0-141-132 kubenswrapper[2567]: E0721 21:07:42.821444 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7138-140442618113200\" is forbidden: User \"system:node:ip-10-0-141-132.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-132.ec2.internal' and this object" podUID="7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b" pod="opendatahub/test-curl-7138-140442618113200" Jul 21 21:07:43.805346 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:43.805317 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b" path="/var/lib/kubelet/pods/7166f3cd-f1e1-49ae-a2fb-0b1f1f64301b/volumes" Jul 21 21:07:43.818424 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:43.818388 2567 generic.go:356] "Generic (PLEG): container finished" podID="a82629e7-1635-417b-bdec-48638d2b1818" containerID="f7efbbe182daad9b5607d28edc68dc74a8053472aaa193ba2e8d7c7338fd99cf" exitCode=0 Jul 21 21:07:43.818575 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:43.818476 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442617865520" event={"ID":"a82629e7-1635-417b-bdec-48638d2b1818","Type":"ContainerDied","Data":"f7efbbe182daad9b5607d28edc68dc74a8053472aaa193ba2e8d7c7338fd99cf"} Jul 21 21:07:44.962883 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:44.962855 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617865520" Jul 21 21:07:44.988045 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:44.988022 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7138-140442617865520_a82629e7-1635-417b-bdec-48638d2b1818/test-curl-7138-140442617865520/0.log" Jul 21 21:07:44.993235 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:44.993214 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td6jp\" (UniqueName: \"kubernetes.io/projected/a82629e7-1635-417b-bdec-48638d2b1818-kube-api-access-td6jp\") pod \"a82629e7-1635-417b-bdec-48638d2b1818\" (UID: \"a82629e7-1635-417b-bdec-48638d2b1818\") " Jul 21 21:07:44.995209 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:44.995184 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82629e7-1635-417b-bdec-48638d2b1818-kube-api-access-td6jp" (OuterVolumeSpecName: "kube-api-access-td6jp") pod "a82629e7-1635-417b-bdec-48638d2b1818" (UID: "a82629e7-1635-417b-bdec-48638d2b1818"). InnerVolumeSpecName "kube-api-access-td6jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:45.022542 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:45.022519 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7138-140442617865520"] Jul 21 21:07:45.033562 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:45.033541 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7138-140442617865520"] Jul 21 21:07:45.094400 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:45.094330 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td6jp\" (UniqueName: \"kubernetes.io/projected/a82629e7-1635-417b-bdec-48638d2b1818-kube-api-access-td6jp\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:45.811728 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:45.811693 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a82629e7-1635-417b-bdec-48638d2b1818" path="/var/lib/kubelet/pods/a82629e7-1635-417b-bdec-48638d2b1818/volumes" Jul 21 21:07:45.831650 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:45.831626 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617865520" Jul 21 21:07:45.831822 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:45.831636 2567 scope.go:117] "RemoveContainer" containerID="f7efbbe182daad9b5607d28edc68dc74a8053472aaa193ba2e8d7c7338fd99cf" Jul 21 21:07:46.782350 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:46.782309 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:07:48.339160 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.339126 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7138-140442617992800"] Jul 21 21:07:48.339572 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.339532 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82629e7-1635-417b-bdec-48638d2b1818" containerName="test-curl-7138-140442617865520" Jul 21 21:07:48.339572 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.339543 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82629e7-1635-417b-bdec-48638d2b1818" containerName="test-curl-7138-140442617865520" Jul 21 21:07:48.339660 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.339620 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82629e7-1635-417b-bdec-48638d2b1818" containerName="test-curl-7138-140442617865520" Jul 21 21:07:48.342699 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.342684 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617992800" Jul 21 21:07:48.345168 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.345144 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pr87k\"" Jul 21 21:07:48.347575 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.347553 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442617992800"] Jul 21 21:07:48.429386 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.429347 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gww8\" (UniqueName: \"kubernetes.io/projected/1a141c38-32ed-4827-828f-d12a387ee13a-kube-api-access-6gww8\") pod \"test-curl-7138-140442617992800\" (UID: \"1a141c38-32ed-4827-828f-d12a387ee13a\") " pod="opendatahub/test-curl-7138-140442617992800" Jul 21 21:07:48.530410 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.530378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gww8\" (UniqueName: \"kubernetes.io/projected/1a141c38-32ed-4827-828f-d12a387ee13a-kube-api-access-6gww8\") pod \"test-curl-7138-140442617992800\" (UID: \"1a141c38-32ed-4827-828f-d12a387ee13a\") " pod="opendatahub/test-curl-7138-140442617992800" Jul 21 21:07:48.539446 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.539419 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gww8\" (UniqueName: \"kubernetes.io/projected/1a141c38-32ed-4827-828f-d12a387ee13a-kube-api-access-6gww8\") pod \"test-curl-7138-140442617992800\" (UID: \"1a141c38-32ed-4827-828f-d12a387ee13a\") " pod="opendatahub/test-curl-7138-140442617992800" Jul 21 21:07:48.655388 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.655354 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617992800" Jul 21 21:07:48.791882 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.791859 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442617992800"] Jul 21 21:07:48.793699 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:07:48.793670 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a141c38_32ed_4827_828f_d12a387ee13a.slice/crio-4da9d12562ab0a122bcc1c54ca8ece8bb37bde3cbd9901ce7b8e586efe17fe11 WatchSource:0}: Error finding container 4da9d12562ab0a122bcc1c54ca8ece8bb37bde3cbd9901ce7b8e586efe17fe11: Status 404 returned error can't find the container with id 4da9d12562ab0a122bcc1c54ca8ece8bb37bde3cbd9901ce7b8e586efe17fe11 Jul 21 21:07:48.852907 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:48.852878 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442617992800" event={"ID":"1a141c38-32ed-4827-828f-d12a387ee13a","Type":"ContainerStarted","Data":"4da9d12562ab0a122bcc1c54ca8ece8bb37bde3cbd9901ce7b8e586efe17fe11"} Jul 21 21:07:49.859874 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:49.859783 2567 generic.go:356] "Generic (PLEG): container finished" podID="1a141c38-32ed-4827-828f-d12a387ee13a" containerID="9b797fb239bbeb0a8577d244071b5374a4d3dc421b4db462751d60e2372057db" exitCode=0 Jul 21 21:07:49.859874 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:49.859838 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442617992800" event={"ID":"1a141c38-32ed-4827-828f-d12a387ee13a","Type":"ContainerDied","Data":"9b797fb239bbeb0a8577d244071b5374a4d3dc421b4db462751d60e2372057db"} Jul 21 21:07:51.005990 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.005927 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617992800" Jul 21 21:07:51.028049 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.028023 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7138-140442617992800_1a141c38-32ed-4827-828f-d12a387ee13a/test-curl-7138-140442617992800/0.log" Jul 21 21:07:51.051383 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.051345 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7138-140442617992800"] Jul 21 21:07:51.054075 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.054047 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gww8\" (UniqueName: \"kubernetes.io/projected/1a141c38-32ed-4827-828f-d12a387ee13a-kube-api-access-6gww8\") pod \"1a141c38-32ed-4827-828f-d12a387ee13a\" (UID: \"1a141c38-32ed-4827-828f-d12a387ee13a\") " Jul 21 21:07:51.055583 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.055561 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7138-140442617992800"] Jul 21 21:07:51.056130 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.056098 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a141c38-32ed-4827-828f-d12a387ee13a-kube-api-access-6gww8" (OuterVolumeSpecName: "kube-api-access-6gww8") pod "1a141c38-32ed-4827-828f-d12a387ee13a" (UID: "1a141c38-32ed-4827-828f-d12a387ee13a"). InnerVolumeSpecName "kube-api-access-6gww8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:51.155096 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.155061 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gww8\" (UniqueName: \"kubernetes.io/projected/1a141c38-32ed-4827-828f-d12a387ee13a-kube-api-access-6gww8\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:51.183507 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.183479 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:51.183881 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.183866 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a141c38-32ed-4827-828f-d12a387ee13a" containerName="test-curl-7138-140442617992800" Jul 21 21:07:51.183924 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.183883 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a141c38-32ed-4827-828f-d12a387ee13a" containerName="test-curl-7138-140442617992800" Jul 21 21:07:51.183992 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.183981 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a141c38-32ed-4827-828f-d12a387ee13a" containerName="test-curl-7138-140442617992800" Jul 21 21:07:51.187000 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.186985 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:51.192117 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.192091 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:51.256301 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.256272 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpkh\" (UniqueName: \"kubernetes.io/projected/4dede3e7-7abd-4f68-bbdd-e6122adcae70-kube-api-access-wfpkh\") pod \"test-curl-7138-140442617994960\" (UID: \"4dede3e7-7abd-4f68-bbdd-e6122adcae70\") " pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:51.357461 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.357429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpkh\" (UniqueName: \"kubernetes.io/projected/4dede3e7-7abd-4f68-bbdd-e6122adcae70-kube-api-access-wfpkh\") pod \"test-curl-7138-140442617994960\" (UID: \"4dede3e7-7abd-4f68-bbdd-e6122adcae70\") " pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:51.366310 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.366285 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpkh\" (UniqueName: \"kubernetes.io/projected/4dede3e7-7abd-4f68-bbdd-e6122adcae70-kube-api-access-wfpkh\") pod \"test-curl-7138-140442617994960\" (UID: \"4dede3e7-7abd-4f68-bbdd-e6122adcae70\") " pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:51.497566 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.497485 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:51.634840 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.634804 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:51.635979 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:07:51.635929 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4dede3e7_7abd_4f68_bbdd_e6122adcae70.slice/crio-956c4b27812f6fafaeb4e3981820b2311c5c2ae070e9e83c02c149ce4f2a796c WatchSource:0}: Error finding container 956c4b27812f6fafaeb4e3981820b2311c5c2ae070e9e83c02c149ce4f2a796c: Status 404 returned error can't find the container with id 956c4b27812f6fafaeb4e3981820b2311c5c2ae070e9e83c02c149ce4f2a796c Jul 21 21:07:51.708015 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.707968 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" podUID="3343a8ed-e9ec-4e00-82b8-f25a85ee718e" containerName="main" containerID="cri-o://ff26cbe81a6a2a8ea8fb867155e9680372576b3a7fbb67b747cadd69fe4661c6" gracePeriod=45 Jul 21 21:07:51.806441 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.806361 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a141c38-32ed-4827-828f-d12a387ee13a" path="/var/lib/kubelet/pods/1a141c38-32ed-4827-828f-d12a387ee13a/volumes" Jul 21 21:07:51.873322 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.873297 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617992800" Jul 21 21:07:51.873322 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.873312 2567 scope.go:117] "RemoveContainer" containerID="9b797fb239bbeb0a8577d244071b5374a4d3dc421b4db462751d60e2372057db" Jul 21 21:07:51.875187 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.875159 2567 generic.go:356] "Generic (PLEG): container finished" podID="3343a8ed-e9ec-4e00-82b8-f25a85ee718e" containerID="ff26cbe81a6a2a8ea8fb867155e9680372576b3a7fbb67b747cadd69fe4661c6" exitCode=0 Jul 21 21:07:51.875357 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.875333 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" event={"ID":"3343a8ed-e9ec-4e00-82b8-f25a85ee718e","Type":"ContainerDied","Data":"ff26cbe81a6a2a8ea8fb867155e9680372576b3a7fbb67b747cadd69fe4661c6"} Jul 21 21:07:51.876760 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.876738 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442617994960" event={"ID":"4dede3e7-7abd-4f68-bbdd-e6122adcae70","Type":"ContainerStarted","Data":"956c4b27812f6fafaeb4e3981820b2311c5c2ae070e9e83c02c149ce4f2a796c"} Jul 21 21:07:51.948683 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:51.948663 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:07:52.064251 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064181 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-home\") pod \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " Jul 21 21:07:52.064701 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064265 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5mld\" (UniqueName: \"kubernetes.io/projected/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-kube-api-access-p5mld\") pod \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " Jul 21 21:07:52.064701 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064286 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tmp-dir\") pod \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " Jul 21 21:07:52.064701 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064307 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-model-cache\") pod \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " Jul 21 21:07:52.064701 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064352 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-dshm\") pod \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " Jul 21 21:07:52.064701 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064407 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tls-certs\") pod \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\" (UID: \"3343a8ed-e9ec-4e00-82b8-f25a85ee718e\") " Jul 21 21:07:52.064701 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064434 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-home" (OuterVolumeSpecName: "home") pod "3343a8ed-e9ec-4e00-82b8-f25a85ee718e" (UID: "3343a8ed-e9ec-4e00-82b8-f25a85ee718e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:52.064701 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064576 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3343a8ed-e9ec-4e00-82b8-f25a85ee718e" (UID: "3343a8ed-e9ec-4e00-82b8-f25a85ee718e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:52.064701 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064597 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3343a8ed-e9ec-4e00-82b8-f25a85ee718e" (UID: "3343a8ed-e9ec-4e00-82b8-f25a85ee718e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:52.065017 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064712 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-home\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:52.065017 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064729 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tmp-dir\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:52.065017 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.064743 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-model-cache\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:52.066448 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.066422 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3343a8ed-e9ec-4e00-82b8-f25a85ee718e" (UID: "3343a8ed-e9ec-4e00-82b8-f25a85ee718e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:07:52.066557 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.066511 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-dshm" (OuterVolumeSpecName: "dshm") pod "3343a8ed-e9ec-4e00-82b8-f25a85ee718e" (UID: "3343a8ed-e9ec-4e00-82b8-f25a85ee718e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:52.066603 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.066549 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-kube-api-access-p5mld" (OuterVolumeSpecName: "kube-api-access-p5mld") pod "3343a8ed-e9ec-4e00-82b8-f25a85ee718e" (UID: "3343a8ed-e9ec-4e00-82b8-f25a85ee718e"). InnerVolumeSpecName "kube-api-access-p5mld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:52.165521 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.165464 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5mld\" (UniqueName: \"kubernetes.io/projected/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-kube-api-access-p5mld\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:52.165521 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.165514 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-dshm\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:52.165521 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.165525 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3343a8ed-e9ec-4e00-82b8-f25a85ee718e-tls-certs\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:52.701384 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.701338 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" podUID="12cd6373-e6c0-480e-a3c0-57d58db46d64" containerName="main" containerID="cri-o://6fa9eb05a4b74611ee490db37b5f7c1e37fb0a7e2e9dd7a77dc2865fcf0a1a51" gracePeriod=45 Jul 21 21:07:52.885916 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.885892 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" Jul 21 21:07:52.886076 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.885923 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw" event={"ID":"3343a8ed-e9ec-4e00-82b8-f25a85ee718e","Type":"ContainerDied","Data":"3f5af60605cc473f3afca4d0502e9c416d548218754338337c02cf022c4beaa6"} Jul 21 21:07:52.886076 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.886013 2567 scope.go:117] "RemoveContainer" containerID="ff26cbe81a6a2a8ea8fb867155e9680372576b3a7fbb67b747cadd69fe4661c6" Jul 21 21:07:52.887696 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.887670 2567 generic.go:356] "Generic (PLEG): container finished" podID="12cd6373-e6c0-480e-a3c0-57d58db46d64" containerID="6fa9eb05a4b74611ee490db37b5f7c1e37fb0a7e2e9dd7a77dc2865fcf0a1a51" exitCode=0 Jul 21 21:07:52.887809 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.887739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" event={"ID":"12cd6373-e6c0-480e-a3c0-57d58db46d64","Type":"ContainerDied","Data":"6fa9eb05a4b74611ee490db37b5f7c1e37fb0a7e2e9dd7a77dc2865fcf0a1a51"} Jul 21 21:07:52.889800 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.889705 2567 generic.go:356] "Generic (PLEG): container finished" podID="4dede3e7-7abd-4f68-bbdd-e6122adcae70" containerID="a576b30b255a43e1e430d3392ed80489d6ae4d1545bcd4427be312fae2fcd945" exitCode=0 Jul 21 21:07:52.889933 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.889794 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442617994960" event={"ID":"4dede3e7-7abd-4f68-bbdd-e6122adcae70","Type":"ContainerDied","Data":"a576b30b255a43e1e430d3392ed80489d6ae4d1545bcd4427be312fae2fcd945"} Jul 21 21:07:52.924603 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.924569 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw"] Jul 21 21:07:52.927459 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.927430 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1ad4ca/rate-test-model-1ad4ca-kserve-5d4cbf44c5-4qsrw"] Jul 21 21:07:52.957672 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:52.957607 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:07:53.074255 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074226 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wvpr\" (UniqueName: \"kubernetes.io/projected/12cd6373-e6c0-480e-a3c0-57d58db46d64-kube-api-access-4wvpr\") pod \"12cd6373-e6c0-480e-a3c0-57d58db46d64\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " Jul 21 21:07:53.074747 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074268 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-dshm\") pod \"12cd6373-e6c0-480e-a3c0-57d58db46d64\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " Jul 21 21:07:53.074747 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074317 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-tmp-dir\") pod \"12cd6373-e6c0-480e-a3c0-57d58db46d64\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " Jul 21 21:07:53.074747 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074435 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-home\") pod \"12cd6373-e6c0-480e-a3c0-57d58db46d64\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " Jul 21 21:07:53.074747 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074530 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd6373-e6c0-480e-a3c0-57d58db46d64-tls-certs\") pod \"12cd6373-e6c0-480e-a3c0-57d58db46d64\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " Jul 21 21:07:53.074747 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074588 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-model-cache\") pod \"12cd6373-e6c0-480e-a3c0-57d58db46d64\" (UID: \"12cd6373-e6c0-480e-a3c0-57d58db46d64\") " Jul 21 21:07:53.074747 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074531 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12cd6373-e6c0-480e-a3c0-57d58db46d64" (UID: "12cd6373-e6c0-480e-a3c0-57d58db46d64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:53.074747 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074694 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-home" (OuterVolumeSpecName: "home") pod "12cd6373-e6c0-480e-a3c0-57d58db46d64" (UID: "12cd6373-e6c0-480e-a3c0-57d58db46d64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:53.075176 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074857 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-model-cache" (OuterVolumeSpecName: "model-cache") pod "12cd6373-e6c0-480e-a3c0-57d58db46d64" (UID: "12cd6373-e6c0-480e-a3c0-57d58db46d64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:53.075176 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074868 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-tmp-dir\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:53.075176 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.074883 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-home\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:53.076496 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.076464 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12cd6373-e6c0-480e-a3c0-57d58db46d64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12cd6373-e6c0-480e-a3c0-57d58db46d64" (UID: "12cd6373-e6c0-480e-a3c0-57d58db46d64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:07:53.076607 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.076513 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-dshm" (OuterVolumeSpecName: "dshm") pod "12cd6373-e6c0-480e-a3c0-57d58db46d64" (UID: "12cd6373-e6c0-480e-a3c0-57d58db46d64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:07:53.076607 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.076558 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cd6373-e6c0-480e-a3c0-57d58db46d64-kube-api-access-4wvpr" (OuterVolumeSpecName: "kube-api-access-4wvpr") pod "12cd6373-e6c0-480e-a3c0-57d58db46d64" (UID: "12cd6373-e6c0-480e-a3c0-57d58db46d64"). InnerVolumeSpecName "kube-api-access-4wvpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:53.175383 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.175352 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wvpr\" (UniqueName: \"kubernetes.io/projected/12cd6373-e6c0-480e-a3c0-57d58db46d64-kube-api-access-4wvpr\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:53.175383 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.175379 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-dshm\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:53.175575 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.175391 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12cd6373-e6c0-480e-a3c0-57d58db46d64-tls-certs\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:53.175575 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.175400 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12cd6373-e6c0-480e-a3c0-57d58db46d64-model-cache\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:53.806306 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.806265 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3343a8ed-e9ec-4e00-82b8-f25a85ee718e" path="/var/lib/kubelet/pods/3343a8ed-e9ec-4e00-82b8-f25a85ee718e/volumes" Jul 21 21:07:53.897742 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.897712 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" Jul 21 21:07:53.897976 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.897760 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7" event={"ID":"12cd6373-e6c0-480e-a3c0-57d58db46d64","Type":"ContainerDied","Data":"2039d3f48c505b5197600d4c1b62812d2bfc1eb955a1fdd396b589e848cea91a"} Jul 21 21:07:53.897976 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.897806 2567 scope.go:117] "RemoveContainer" containerID="6fa9eb05a4b74611ee490db37b5f7c1e37fb0a7e2e9dd7a77dc2865fcf0a1a51" Jul 21 21:07:53.915419 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.915390 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7"] Jul 21 21:07:53.921384 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:53.921350 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9b964f/rate-test-model-9b964f-kserve-5589f6b88c-tjpt7"] Jul 21 21:07:54.051198 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.051174 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:54.072930 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.072873 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7138-140442617994960_4dede3e7-7abd-4f68-bbdd-e6122adcae70/test-curl-7138-140442617994960/0.log" Jul 21 21:07:54.098010 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.097986 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:54.102605 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.102580 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:54.183689 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.183659 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfpkh\" (UniqueName: \"kubernetes.io/projected/4dede3e7-7abd-4f68-bbdd-e6122adcae70-kube-api-access-wfpkh\") pod \"4dede3e7-7abd-4f68-bbdd-e6122adcae70\" (UID: \"4dede3e7-7abd-4f68-bbdd-e6122adcae70\") " Jul 21 21:07:54.185761 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.185737 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dede3e7-7abd-4f68-bbdd-e6122adcae70-kube-api-access-wfpkh" (OuterVolumeSpecName: "kube-api-access-wfpkh") pod "4dede3e7-7abd-4f68-bbdd-e6122adcae70" (UID: "4dede3e7-7abd-4f68-bbdd-e6122adcae70"). InnerVolumeSpecName "kube-api-access-wfpkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:54.225625 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.225594 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:54.226043 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.226028 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dede3e7-7abd-4f68-bbdd-e6122adcae70" containerName="test-curl-7138-140442617994960" Jul 21 21:07:54.226086 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.226045 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dede3e7-7abd-4f68-bbdd-e6122adcae70" containerName="test-curl-7138-140442617994960" Jul 21 21:07:54.226086 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.226056 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3343a8ed-e9ec-4e00-82b8-f25a85ee718e" containerName="main" Jul 21 21:07:54.226086 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.226063 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3343a8ed-e9ec-4e00-82b8-f25a85ee718e" containerName="main" Jul 21 21:07:54.226086 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.226085 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12cd6373-e6c0-480e-a3c0-57d58db46d64" containerName="main" Jul 21 21:07:54.226226 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.226091 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12cd6373-e6c0-480e-a3c0-57d58db46d64" containerName="main" Jul 21 21:07:54.226226 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.226157 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dede3e7-7abd-4f68-bbdd-e6122adcae70" containerName="test-curl-7138-140442617994960" Jul 21 21:07:54.226226 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.226168 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3343a8ed-e9ec-4e00-82b8-f25a85ee718e" containerName="main" Jul 21 21:07:54.226226 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.226174 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="12cd6373-e6c0-480e-a3c0-57d58db46d64" containerName="main" Jul 21 21:07:54.229255 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.229230 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:54.234595 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.234571 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:54.285168 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.285136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtnj\" (UniqueName: \"kubernetes.io/projected/0a890270-5928-4f7b-b109-14c9442373ae-kube-api-access-2xtnj\") pod \"test-curl-7138-140442617994960\" (UID: \"0a890270-5928-4f7b-b109-14c9442373ae\") " pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:54.285339 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.285288 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfpkh\" (UniqueName: \"kubernetes.io/projected/4dede3e7-7abd-4f68-bbdd-e6122adcae70-kube-api-access-wfpkh\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:54.386275 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.386243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtnj\" (UniqueName: \"kubernetes.io/projected/0a890270-5928-4f7b-b109-14c9442373ae-kube-api-access-2xtnj\") pod \"test-curl-7138-140442617994960\" (UID: \"0a890270-5928-4f7b-b109-14c9442373ae\") " pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:54.395154 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.395121 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtnj\" (UniqueName: \"kubernetes.io/projected/0a890270-5928-4f7b-b109-14c9442373ae-kube-api-access-2xtnj\") pod \"test-curl-7138-140442617994960\" (UID: \"0a890270-5928-4f7b-b109-14c9442373ae\") " pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:54.540312 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.540282 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:54.676286 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.676255 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:54.677935 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:07:54.677904 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a890270_5928_4f7b_b109_14c9442373ae.slice/crio-2ea0d05433d5d340418bdb3d0047af19e8dfb230a5031af19efec6ad233b487f WatchSource:0}: Error finding container 2ea0d05433d5d340418bdb3d0047af19e8dfb230a5031af19efec6ad233b487f: Status 404 returned error can't find the container with id 2ea0d05433d5d340418bdb3d0047af19e8dfb230a5031af19efec6ad233b487f Jul 21 21:07:54.905486 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.905455 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442617994960" event={"ID":"0a890270-5928-4f7b-b109-14c9442373ae","Type":"ContainerStarted","Data":"2ea0d05433d5d340418bdb3d0047af19e8dfb230a5031af19efec6ad233b487f"} Jul 21 21:07:54.906606 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.906587 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:54.906606 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.906605 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="956c4b27812f6fafaeb4e3981820b2311c5c2ae070e9e83c02c149ce4f2a796c" Jul 21 21:07:54.918734 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:54.918703 2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7138-140442617994960" oldPodUID="4dede3e7-7abd-4f68-bbdd-e6122adcae70" podUID="0a890270-5928-4f7b-b109-14c9442373ae" Jul 21 21:07:55.806275 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:55.806235 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12cd6373-e6c0-480e-a3c0-57d58db46d64" path="/var/lib/kubelet/pods/12cd6373-e6c0-480e-a3c0-57d58db46d64/volumes" Jul 21 21:07:55.806762 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:55.806731 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dede3e7-7abd-4f68-bbdd-e6122adcae70" path="/var/lib/kubelet/pods/4dede3e7-7abd-4f68-bbdd-e6122adcae70/volumes" Jul 21 21:07:55.914439 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:55.914402 2567 generic.go:356] "Generic (PLEG): container finished" podID="0a890270-5928-4f7b-b109-14c9442373ae" containerID="a2ae953be6607cfc0cbd8140384d06b7964df711e940549773d27d8093daf042" exitCode=0 Jul 21 21:07:55.914633 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:55.914489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7138-140442617994960" event={"ID":"0a890270-5928-4f7b-b109-14c9442373ae","Type":"ContainerDied","Data":"a2ae953be6607cfc0cbd8140384d06b7964df711e940549773d27d8093daf042"} Jul 21 21:07:57.055521 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.055497 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:57.076675 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.076652 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7138-140442617994960_0a890270-5928-4f7b-b109-14c9442373ae/test-curl-7138-140442617994960/0.log" Jul 21 21:07:57.100869 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.100841 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:57.104817 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.104795 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7138-140442617994960"] Jul 21 21:07:57.113442 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.113425 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xtnj\" (UniqueName: \"kubernetes.io/projected/0a890270-5928-4f7b-b109-14c9442373ae-kube-api-access-2xtnj\") pod \"0a890270-5928-4f7b-b109-14c9442373ae\" (UID: \"0a890270-5928-4f7b-b109-14c9442373ae\") " Jul 21 21:07:57.115462 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.115439 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a890270-5928-4f7b-b109-14c9442373ae-kube-api-access-2xtnj" (OuterVolumeSpecName: "kube-api-access-2xtnj") pod "0a890270-5928-4f7b-b109-14c9442373ae" (UID: "0a890270-5928-4f7b-b109-14c9442373ae"). InnerVolumeSpecName "kube-api-access-2xtnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:57.215005 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.214923 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtnj\" (UniqueName: \"kubernetes.io/projected/0a890270-5928-4f7b-b109-14c9442373ae-kube-api-access-2xtnj\") on node \"ip-10-0-141-132.ec2.internal\" DevicePath \"\"" Jul 21 21:07:57.805261 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.805231 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a890270-5928-4f7b-b109-14c9442373ae" path="/var/lib/kubelet/pods/0a890270-5928-4f7b-b109-14c9442373ae/volumes" Jul 21 21:07:57.927464 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.927408 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7138-140442617994960" Jul 21 21:07:57.927464 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:07:57.927448 2567 scope.go:117] "RemoveContainer" containerID="a2ae953be6607cfc0cbd8140384d06b7964df711e940549773d27d8093daf042" Jul 21 21:08:07.392279 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:07.392244 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:08:11.641693 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:11.641657 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79979d5c99-hzkmt_cdf0d01d-f88a-40d3-bfa2-3f862001eb64/authorino/0.log" Jul 21 21:08:17.723829 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:17.723799 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb_90c84641-8fdc-452c-93b8-d1fe8b7d2f3c/copy-manifests/0.log" Jul 21 21:08:17.730485 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:17.730466 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb_90c84641-8fdc-452c-93b8-d1fe8b7d2f3c/manager/0.log" Jul 21 21:08:18.851153 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:18.851105 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-api-b-280285-7d684f646c-cs72d_0610435f-c1d8-4b93-af83-1618fb875c77/maas-api/0.log" Jul 21 21:08:19.094751 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:19.094700 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-2ba36e-76495dd99f-wp9gx_cb6fb955-805e-40c1-ac1b-0d996d196476/maas-api/0.log" Jul 21 21:08:20.380496 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:20.380467 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-ait-7bc6ce31-29744460-4n6h2_435b529f-0961-4e4f-92da-3db61135f8ab/cleanup/0.log" Jul 21 21:08:20.493745 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:20.493713 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-derive-3f69897a-29744460-7xpzj_8862c886-7e33-4048-9061-6decc23738be/cleanup/0.log" Jul 21 21:08:20.877768 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:20.877730 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-bh898"] Jul 21 21:08:21.915813 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:21.915778 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7_a60a7b4d-aa89-48e5-bc0c-6dfcca412956/util/0.log" Jul 21 21:08:21.922745 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:21.922723 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7_a60a7b4d-aa89-48e5-bc0c-6dfcca412956/pull/0.log" Jul 21 21:08:21.928618 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:21.928598 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7_a60a7b4d-aa89-48e5-bc0c-6dfcca412956/extract/0.log" Jul 21 21:08:22.039955 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.039924 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k_966e8087-e229-4c1e-8011-c61e3e8b2b15/util/0.log" Jul 21 21:08:22.045759 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.045737 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k_966e8087-e229-4c1e-8011-c61e3e8b2b15/pull/0.log" Jul 21 21:08:22.051416 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.051394 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k_966e8087-e229-4c1e-8011-c61e3e8b2b15/extract/0.log" Jul 21 21:08:22.160288 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.160259 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8_12c01489-998c-4a0f-b70e-a80c9e05c43f/util/0.log" Jul 21 21:08:22.166867 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.166807 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8_12c01489-998c-4a0f-b70e-a80c9e05c43f/pull/0.log" Jul 21 21:08:22.172998 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.172975 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8_12c01489-998c-4a0f-b70e-a80c9e05c43f/extract/0.log" Jul 21 21:08:22.281229 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.281192 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf_d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b/util/0.log" Jul 21 21:08:22.286988 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.286966 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf_d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b/pull/0.log" Jul 21 21:08:22.292431 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.292414 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf_d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b/extract/0.log" Jul 21 21:08:22.404925 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:22.404898 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79979d5c99-hzkmt_cdf0d01d-f88a-40d3-bfa2-3f862001eb64/authorino/0.log" Jul 21 21:08:23.092166 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:23.092136 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-bh898_62c1acb0-bd2e-49f5-866a-29fbb63ff00c/limitador/0.log" Jul 21 21:08:23.903902 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:23.903871 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567c747899-w859j_52cb6c66-80e9-40cd-a37d-fd57cba871a0/kube-auth-proxy/0.log" Jul 21 21:08:24.888483 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:24.888451 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-cqltf_5c628870-82ea-42e7-b807-a28c5c1ff7e1/main/0.log" Jul 21 21:08:25.139307 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:25.139230 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f96qc2_97d3a1e6-773c-4d14-819e-f27235042c2f/main/0.log" Jul 21 21:08:30.391035 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.390998 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jgpxl/must-gather-gs9zs"] Jul 21 21:08:30.391420 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.391375 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a890270-5928-4f7b-b109-14c9442373ae" containerName="test-curl-7138-140442617994960" Jul 21 21:08:30.391420 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.391386 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a890270-5928-4f7b-b109-14c9442373ae" containerName="test-curl-7138-140442617994960" Jul 21 21:08:30.391499 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.391466 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a890270-5928-4f7b-b109-14c9442373ae" containerName="test-curl-7138-140442617994960" Jul 21 21:08:30.394882 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.394861 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jgpxl/must-gather-gs9zs" Jul 21 21:08:30.397614 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.397595 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jgpxl\"/\"kube-root-ca.crt\"" Jul 21 21:08:30.397736 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.397650 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jgpxl\"/\"openshift-service-ca.crt\"" Jul 21 21:08:30.404164 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.404137 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jgpxl/must-gather-gs9zs"] Jul 21 21:08:30.522658 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.522624 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38639613-53e1-4109-ae28-7c44a36c7a1a-must-gather-output\") pod \"must-gather-gs9zs\" (UID: \"38639613-53e1-4109-ae28-7c44a36c7a1a\") " pod="openshift-must-gather-jgpxl/must-gather-gs9zs" Jul 21 21:08:30.522830 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.522670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bg9k\" (UniqueName: \"kubernetes.io/projected/38639613-53e1-4109-ae28-7c44a36c7a1a-kube-api-access-6bg9k\") pod \"must-gather-gs9zs\" (UID: \"38639613-53e1-4109-ae28-7c44a36c7a1a\") " pod="openshift-must-gather-jgpxl/must-gather-gs9zs" Jul 21 21:08:30.623438 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.623401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38639613-53e1-4109-ae28-7c44a36c7a1a-must-gather-output\") pod \"must-gather-gs9zs\" (UID: \"38639613-53e1-4109-ae28-7c44a36c7a1a\") " pod="openshift-must-gather-jgpxl/must-gather-gs9zs" Jul 21 21:08:30.623647 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.623446 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bg9k\" (UniqueName: \"kubernetes.io/projected/38639613-53e1-4109-ae28-7c44a36c7a1a-kube-api-access-6bg9k\") pod \"must-gather-gs9zs\" (UID: \"38639613-53e1-4109-ae28-7c44a36c7a1a\") " pod="openshift-must-gather-jgpxl/must-gather-gs9zs" Jul 21 21:08:30.623809 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.623786 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38639613-53e1-4109-ae28-7c44a36c7a1a-must-gather-output\") pod \"must-gather-gs9zs\" (UID: \"38639613-53e1-4109-ae28-7c44a36c7a1a\") " pod="openshift-must-gather-jgpxl/must-gather-gs9zs" Jul 21 21:08:30.632929 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.632902 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bg9k\" (UniqueName: \"kubernetes.io/projected/38639613-53e1-4109-ae28-7c44a36c7a1a-kube-api-access-6bg9k\") pod \"must-gather-gs9zs\" (UID: \"38639613-53e1-4109-ae28-7c44a36c7a1a\") " pod="openshift-must-gather-jgpxl/must-gather-gs9zs" Jul 21 21:08:30.704498 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.704406 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jgpxl/must-gather-gs9zs" Jul 21 21:08:30.847696 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:30.847670 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jgpxl/must-gather-gs9zs"] Jul 21 21:08:30.849931 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:08:30.849900 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38639613_53e1_4109_ae28_7c44a36c7a1a.slice/crio-9403c4b3aae786cdaaa50c06ad12a2e468b40243816526f8e85cecb94d10d676 WatchSource:0}: Error finding container 9403c4b3aae786cdaaa50c06ad12a2e468b40243816526f8e85cecb94d10d676: Status 404 returned error can't find the container with id 9403c4b3aae786cdaaa50c06ad12a2e468b40243816526f8e85cecb94d10d676 Jul 21 21:08:31.122196 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:31.122164 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jgpxl/must-gather-gs9zs" event={"ID":"38639613-53e1-4109-ae28-7c44a36c7a1a","Type":"ContainerStarted","Data":"9403c4b3aae786cdaaa50c06ad12a2e468b40243816526f8e85cecb94d10d676"} Jul 21 21:08:32.130037 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:32.129996 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jgpxl/must-gather-gs9zs" event={"ID":"38639613-53e1-4109-ae28-7c44a36c7a1a","Type":"ContainerStarted","Data":"233dec18f17e208a339e0425196c24959adc70b8d8424d57e6c9852295dbc2c6"} Jul 21 21:08:32.130037 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:32.130044 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jgpxl/must-gather-gs9zs" event={"ID":"38639613-53e1-4109-ae28-7c44a36c7a1a","Type":"ContainerStarted","Data":"ebedcb7c238ce5bca9686810419632f6580043254f8db2f820f22c6f5be5d115"} Jul 21 21:08:32.148051 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:32.147998 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jgpxl/must-gather-gs9zs" podStartSLOduration=1.393476536 podStartE2EDuration="2.147982552s" totalImagesPullingTime="754.506016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:32.146556308 +0000 UTC m=+3266.826682766" watchObservedRunningTime="2026-07-21 21:08:32.147982552 +0000 UTC m=+3266.828108992" Jul 21 21:08:33.366378 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:33.366348 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zps6q_ec48f297-40e1-4b50-a162-d46157ba43da/global-pull-secret-syncer/0.log" Jul 21 21:08:33.460746 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:33.460710 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rwgp8_5e2eb5f8-a554-4bd8-8c85-0aa097e3e7e1/konnectivity-agent/0.log" Jul 21 21:08:33.556047 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:33.556008 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-132.ec2.internal_bfc1999bccbcd5a41204f2e93a96d415/haproxy/0.log" Jul 21 21:08:38.695756 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.695703 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7_a60a7b4d-aa89-48e5-bc0c-6dfcca412956/extract/0.log" Jul 21 21:08:38.720005 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.719920 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7_a60a7b4d-aa89-48e5-bc0c-6dfcca412956/util/0.log" Jul 21 21:08:38.746250 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.746219 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7598pcm7_a60a7b4d-aa89-48e5-bc0c-6dfcca412956/pull/0.log" Jul 21 21:08:38.792179 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.792083 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k_966e8087-e229-4c1e-8011-c61e3e8b2b15/extract/0.log" Jul 21 21:08:38.818453 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.818425 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k_966e8087-e229-4c1e-8011-c61e3e8b2b15/util/0.log" Jul 21 21:08:38.846492 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.846466 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c749k_966e8087-e229-4c1e-8011-c61e3e8b2b15/pull/0.log" Jul 21 21:08:38.883839 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.883811 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8_12c01489-998c-4a0f-b70e-a80c9e05c43f/extract/0.log" Jul 21 21:08:38.921760 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.921716 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8_12c01489-998c-4a0f-b70e-a80c9e05c43f/util/0.log" Jul 21 21:08:38.949094 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.949022 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73g2gz8_12c01489-998c-4a0f-b70e-a80c9e05c43f/pull/0.log" Jul 21 21:08:38.986810 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:38.986755 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf_d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b/extract/0.log" Jul 21 21:08:39.021252 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:39.021225 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf_d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b/util/0.log" Jul 21 21:08:39.047736 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:39.047703 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4ttf_d2e0ddd3-0a2c-4ec8-ac0d-fb3faeebb66b/pull/0.log" Jul 21 21:08:39.499204 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:39.499167 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79979d5c99-hzkmt_cdf0d01d-f88a-40d3-bfa2-3f862001eb64/authorino/0.log" Jul 21 21:08:39.912885 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:39.912849 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-bh898_62c1acb0-bd2e-49f5-866a-29fbb63ff00c/limitador/0.log" Jul 21 21:08:41.749173 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:41.749133 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-r5j5b_0af8ce4b-1621-4236-87e4-3705792ab31b/cluster-monitoring-operator/0.log" Jul 21 21:08:41.779303 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:41.779248 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-ccqsj_1a8b1b66-cf08-4305-a05e-82a17c308318/kube-state-metrics/0.log" Jul 21 21:08:41.801265 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:41.801232 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-ccqsj_1a8b1b66-cf08-4305-a05e-82a17c308318/kube-rbac-proxy-main/0.log" Jul 21 21:08:41.828264 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:41.828240 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-ccqsj_1a8b1b66-cf08-4305-a05e-82a17c308318/kube-rbac-proxy-self/0.log" Jul 21 21:08:41.858560 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:41.858530 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54ff9d6fdf-gmm5p_03255225-ed9a-4f13-a6e9-657af5a34306/metrics-server/0.log" Jul 21 21:08:42.002998 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.002892 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pql9x_8349de3d-1a4c-42fe-bdba-ee4d76730314/node-exporter/0.log" Jul 21 21:08:42.024989 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.024930 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pql9x_8349de3d-1a4c-42fe-bdba-ee4d76730314/kube-rbac-proxy/0.log" Jul 21 21:08:42.049232 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.049200 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pql9x_8349de3d-1a4c-42fe-bdba-ee4d76730314/init-textfile/0.log" Jul 21 21:08:42.158521 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.158486 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-6p4g4_6921272b-a355-44c3-a0c7-73b972bbee34/kube-rbac-proxy-main/0.log" Jul 21 21:08:42.183251 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.183202 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-6p4g4_6921272b-a355-44c3-a0c7-73b972bbee34/kube-rbac-proxy-self/0.log" Jul 21 21:08:42.207353 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.207312 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-6p4g4_6921272b-a355-44c3-a0c7-73b972bbee34/openshift-state-metrics/0.log" Jul 21 21:08:42.450549 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.450519 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-wl8dl_0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4/prometheus-operator/0.log" Jul 21 21:08:42.477506 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.477478 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-wl8dl_0f22e398-ffef-42d5-8aeb-00a5cf4d5bd4/kube-rbac-proxy/0.log" Jul 21 21:08:42.502276 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.502247 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-l7jzc_ca5de520-0d52-4211-843e-ab7dc0ec2ea0/prometheus-operator-admission-webhook/0.log" Jul 21 21:08:42.539185 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.539153 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d4bd6fd4b-9wmff_9c44d643-91df-41d0-9a33-f27572987b54/telemeter-client/0.log" Jul 21 21:08:42.563194 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.563167 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d4bd6fd4b-9wmff_9c44d643-91df-41d0-9a33-f27572987b54/reload/0.log" Jul 21 21:08:42.587407 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:42.587374 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d4bd6fd4b-9wmff_9c44d643-91df-41d0-9a33-f27572987b54/kube-rbac-proxy/0.log" Jul 21 21:08:43.369412 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.369376 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72"] Jul 21 21:08:43.379149 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.379116 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.382807 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.382785 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jgpxl\"/\"default-dockercfg-kgt6b\"" Jul 21 21:08:43.384663 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.384632 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72"] Jul 21 21:08:43.475459 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.472863 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-proc\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.475459 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.473000 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-lib-modules\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.475459 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.473053 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-podres\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.475459 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.473130 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jft\" (UniqueName: \"kubernetes.io/projected/23847340-62a6-416a-a23a-1ec76818ae22-kube-api-access-w7jft\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.475459 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.473212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-sys\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.574357 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.574305 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-proc\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.574554 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.574378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-lib-modules\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.574554 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.574408 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-podres\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.574554 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.574455 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jft\" (UniqueName: \"kubernetes.io/projected/23847340-62a6-416a-a23a-1ec76818ae22-kube-api-access-w7jft\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.574554 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.574506 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-sys\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.574782 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.574627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-sys\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.574782 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.574696 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-proc\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.574892 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.574847 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-lib-modules\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.575000 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.574980 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23847340-62a6-416a-a23a-1ec76818ae22-podres\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.587249 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.587221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jft\" (UniqueName: \"kubernetes.io/projected/23847340-62a6-416a-a23a-1ec76818ae22-kube-api-access-w7jft\") pod \"perf-node-gather-daemonset-w6g72\" (UID: \"23847340-62a6-416a-a23a-1ec76818ae22\") " pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.696127 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.695842 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:43.896339 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.896304 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72"] Jul 21 21:08:43.900990 ip-10-0-141-132 kubenswrapper[2567]: W0721 21:08:43.900956 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23847340_62a6_416a_a23a_1ec76818ae22.slice/crio-5debee2b226659fb19b6b1095bb42a1141f42c49073aa29292e52589d8d5e63a WatchSource:0}: Error finding container 5debee2b226659fb19b6b1095bb42a1141f42c49073aa29292e52589d8d5e63a: Status 404 returned error can't find the container with id 5debee2b226659fb19b6b1095bb42a1141f42c49073aa29292e52589d8d5e63a Jul 21 21:08:43.933604 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:43.933581 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-vddm7_c069c98e-f0ef-43cd-a1b1-bbda4e3a0eb8/networking-console-plugin/0.log" Jul 21 21:08:44.249517 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:44.249423 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" event={"ID":"23847340-62a6-416a-a23a-1ec76818ae22","Type":"ContainerStarted","Data":"a6215f78eb62286818f87266df5ee51b9a8ca31a9f7f0321739bdb5abb07ad81"} Jul 21 21:08:44.249517 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:44.249490 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" event={"ID":"23847340-62a6-416a-a23a-1ec76818ae22","Type":"ContainerStarted","Data":"5debee2b226659fb19b6b1095bb42a1141f42c49073aa29292e52589d8d5e63a"} Jul 21 21:08:44.249704 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:44.249527 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:44.268260 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:44.268213 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" podStartSLOduration=1.268196137 podStartE2EDuration="1.268196137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:44.266018269 +0000 UTC m=+3278.946144735" watchObservedRunningTime="2026-07-21 21:08:44.268196137 +0000 UTC m=+3278.948322574" Jul 21 21:08:44.964924 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:44.964892 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-846bf86f88-8g5vp_61e462c0-8f16-405e-8f0c-ae8f9929e315/console/0.log" Jul 21 21:08:45.626220 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:45.626186 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-hbsv4_9c0b6228-cec1-498a-8bef-bd0087d6d153/volume-data-source-validator/0.log" Jul 21 21:08:46.718033 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:46.717997 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g29mm_9bde1aba-deb0-4a77-898b-d3a48bcfca67/dns/0.log" Jul 21 21:08:46.742633 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:46.742604 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g29mm_9bde1aba-deb0-4a77-898b-d3a48bcfca67/kube-rbac-proxy/0.log" Jul 21 21:08:46.852195 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:46.852165 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jfpqq_6e05dd41-bfb8-4bd8-bfe4-955dce5591f4/dns-node-resolver/0.log" Jul 21 21:08:47.518477 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:47.518445 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4dc2n_32f0ea72-8c4a-4787-bb7f-b6aa0991eef2/node-ca/0.log" Jul 21 21:08:48.794836 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:48.794810 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567c747899-w859j_52cb6c66-80e9-40cd-a37d-fd57cba871a0/kube-auth-proxy/0.log" Jul 21 21:08:49.648406 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:49.648373 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-czv2k_a3190751-b8b6-4693-a0bd-ff461895f8a6/serve-healthcheck-canary/0.log" Jul 21 21:08:50.199153 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:50.199121 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-95dqt_4aec0aca-f346-46fb-8392-24e10c582da2/insights-operator/0.log" Jul 21 21:08:50.199873 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:50.199853 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-95dqt_4aec0aca-f346-46fb-8392-24e10c582da2/insights-operator/1.log" Jul 21 21:08:50.222746 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:50.222715 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-57t7d_48f4c58e-85c1-435a-bd62-30c0950be9ba/kube-rbac-proxy/0.log" Jul 21 21:08:50.248685 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:50.248664 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-57t7d_48f4c58e-85c1-435a-bd62-30c0950be9ba/exporter/0.log" Jul 21 21:08:50.269188 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:50.269156 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jgpxl/perf-node-gather-daemonset-w6g72" Jul 21 21:08:50.272163 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:50.272138 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-57t7d_48f4c58e-85c1-435a-bd62-30c0950be9ba/extractor/0.log" Jul 21 21:08:53.357558 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:53.357527 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb_90c84641-8fdc-452c-93b8-d1fe8b7d2f3c/manager/0.log" Jul 21 21:08:53.387314 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:53.387284 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b9bbc76fb-2n5sb_90c84641-8fdc-452c-93b8-d1fe8b7d2f3c/copy-manifests/0.log" Jul 21 21:08:54.399214 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:54.399187 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567c747899-w859j_52cb6c66-80e9-40cd-a37d-fd57cba871a0/kube-auth-proxy/0.log" Jul 21 21:08:54.856653 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:54.856627 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-697fc85fc4-7mtgp_bdc0f0fe-adf9-418d-bfbb-2e1034263fa7/manager/0.log" Jul 21 21:08:54.944252 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:54.944171 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-h7v72_23ed2c97-e359-4f03-9c62-abaac45193d3/openshift-lws-operator/0.log" Jul 21 21:08:56.046897 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:56.046865 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-567c747899-w859j_52cb6c66-80e9-40cd-a37d-fd57cba871a0/kube-auth-proxy/0.log" Jul 21 21:08:59.854673 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:59.854642 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-qwp4n_afc7da83-8a9f-4118-93a6-d55193041472/migrator/0.log" Jul 21 21:08:59.880764 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:08:59.880739 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-qwp4n_afc7da83-8a9f-4118-93a6-d55193041472/graceful-termination/0.log" Jul 21 21:09:00.260868 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:09:00.260830 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-66qg5_3b28e11a-e4ce-4b63-966d-60864726d9fc/kube-storage-version-migrator-operator/1.log" Jul 21 21:09:00.261996 ip-10-0-141-132 kubenswrapper[2567]: I0721 21:09:00.261970 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-66qg5_3b28e11a-e4ce-4b63-966d-60864726d9fc/kube-storage-version-migrator-operator/0.log"